Embedded Postgres in Go

Transcript

1. Embedded Postgres in Go Golang Warsaw #67 (spring) 2026 Oleg Kovalov 󰑒 olegk.dev
2. - Intro - Problem - Solution - 1 weekend - Conclusions Agenda 2
3. Me 3
4. - Open-source addicted gopher Me 4
5. - Open-source addicted gopher - Fan of linters (co-author go-critic) Me 5
6. - Open-source addicted gopher - Fan of linters (co-author go-critic) - 2 labradors 💛🖤 Me 6
7. - Open-source addicted gopher - Fan of linters (co-author go-critic) - 2 labradors 💛🖤 - Nearmap engineer Me 7
8. - Open-source addicted gopher - Fan of linters (co-author go-critic) - 2 labradors 💛🖤 - Nearmap engineer - ... olegk.dev Me 8
9. Long story short 9 short story long vs
10. 10 I have only 1.5 stories for today
11. Intro 11
12. - Definitely not at Nearmap Intro 12
13. - Definitely not at Nearmap - which is freaking good Intro 13
14. - Definitely not at Nearmap - which is freaking good - we’re hiring btw Intro 14
15. - Definitely not at Nearmap - which is freaking good - we’re hiring btw - One of my previous employers Intro 15
16. - Definitely not at Nearmap - which is freaking good - we’re hiring btw - One of my previous employers - thankfully I’m not here anymore Intro 16
17. - Definitely not at Nearmap - which is freaking good - we’re hiring btw - One of my previous employers - thankfully I’m not here anymore - Never over complicate your infrastructure Intro 17
18. - Definitely not at Nearmap - which is freaking good - we’re hiring btw - One of my previous employers - thankfully I’m not here anymore - Never over complicate your infrastructure - especially CI Intro 18
19. - Definitely not at Nearmap - which is freaking good - we’re hiring btw - One of my previous employers - thankfully I’m not here anymore - Never over complicate your infrastructure - especially CI - Okay, let’s see how deep is rabbit hole Intro 19
20. - Definitely not at Nearmap - which is freaking good - we’re hiring btw - One of my previous employers - thankfully I’m not here anymore - Never over complicate your infrastructure - especially CI - Okay, let’s see how deep is rabbit hole - or 💩hole Intro 20
21. Problem statement 21
22. - Go Problem statement 22
23. - Go - Postgres Problem statement 23
24. - Go - Postgres - Monorepo Problem statement 24
25. - Go - Postgres - Monorepo - Bazel Problem statement 25
26. - Go - Postgres - Monorepo - Bazel - Engineers with AI spice with unlimited tokens Problem statement 26
27. Not bad, not terrible 27
28. Not bad, not terrible 28 Prod DB (Postgres) Service (Go)
29. Not bad, not terrible 29 DB (Postgres) Prod Canary DB (Postgres) Service (Go) Service (Go)
30. Not bad, not terrible 30 DB (Postgres) Prod Canary CI Local DB (Postgres) DB (Postgres) Service (Go) Service (Go) DB (Postgres) Service (Go) Service (Go)
31. Not bad, not terrible 31 DB (Postgres) Prod Canary CI Local DB (Postgres) DB (Postgres) DB (Postgres) Service (Go) Service (Go) Service (Go) Service (Go) BORING BORING BORING BORING BORING
32. Software engineering 32
33. Software engineering 33 DB (Postgres) Prod Canary CI Local DB (Postgres) DB (Postgres) DB (Postgres) Service (Go) Service (Go) Service (Go) Service (Go)
34. Software engineering 34 DB (Postgres) Prod Canary CI Local DB (SQLite) DB (Postgres) DB (Postgres) Service (Go) Service (Go) Service (Go) Service (Go) SQLite
35. Software engineering, for real 35 DB (Postgres) Prod Canary CI Local DB (SQLite) DB (Postgres) DB (SQLite) Service (Go) Service (Go) Service (Go) Service (Go) SQLite SQLite
36. Software engineering happened 36 DB (Postgres) Prod Canary CI Local DB (SQLite) DB (Postgres) DB (SQLite) Service (Go) Service (Go) Service (Go) Service (Go) SQLite SQLite
37. *Slide-filler about strange decisions* 37
38. Common sense 38
39. - Use local Postgres Common sense 39
40. - Use local Postgres - easy but might have side-effects Common sense 40
41. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local Common sense 41
42. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local - more complicated but more powerful Common sense 42
43. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local - more complicated but more powerful - but even more side-effects Common sense 43
44. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local - more complicated but more powerful - but even more side-effects - Use Docker Common sense 44
45. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local - more complicated but more powerful - but even more side-effects - Use Docker - a bit of routine but for 2025 is not a problem Common sense 45
46. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local - more complicated but more powerful - but even more side-effects - Use Docker - a bit of routine but for 2025 is not a problem - no idea why not use it tbh Common sense 46
47. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local - more complicated but more powerful - but even more side-effects - Use Docker - a bit of routine but for 2025 is not a problem - no idea why not use it tbh - WASM Common sense 47
48. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local - more complicated but more powerful - but even more side-effects - Use Docker - a bit of routine but for 2025 is not a problem - no idea why not use it tbh - WASM - modern-ish, limited, but interesting Common sense 48
49. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local - more complicated but more powerful - but even more side-effects - Use Docker - a bit of routine but for 2025 is not a problem - no idea why not use it tbh - WASM - modern-ish, limited, but interesting - Use…embedded Postgres Common sense 49
50. - Use local Postgres - easy but might have side-effects - Use remote Postgres for CI or local - more complicated but more powerful - but even more side-effects - Use Docker - a bit of routine but for 2025 is not a problem - no idea why not use it tbh - WASM - modern-ish, limited, but interesting - Use…embedded Postgres - wat? Common sense 50
51. Embedded Postgres 51
52. - Instead of shipping a Docker just ship a Postgres binary - pre-Docker era, isn’t? Embedded Postgres 52
53. - Instead of shipping a Docker just ship a Postgres binary - pre-Docker era, isn’t? - Nothing revolutionary - and no extremism as well Embedded Postgres 53
54. - Instead of shipping a Docker just ship a Postgres binary - pre-Docker era, isn’t? - Nothing revolutionary - and no extremism as well - Postgres is stable - best software ever? Embedded Postgres 54
55. - Instead of shipping a Docker just ship a Postgres binary - pre-Docker era, isn’t? - Nothing revolutionary - and no extremism as well - Postgres is stable - best software ever? - Dozens of solutions already - JS: https://www.npmjs.com/package/embedded-postgres - Rust: https://github.com/theseus-rs/postgresql-embedded - Java: https://github.com/zonkyio/embedded-postgres - Python: https://github.com/orm011/pgserver - Go: https://github.com/fergusstrange/embedded-postgres - … Embedded Postgres 55
56. How it works? 56
57. - Download How it works? 57
58. - Download - Unpack How it works? 58
59. - Download - Unpack - Start How it works? 59
60. - Download - Unpack - Start - … How it works? 60
61. - Download - Unpack - Start - … - PROFIT How it works? 61
62. - Download - Unpack - Start - … - PROFIT It’s still a Postgres. How it works? 62
63. - Download - Unpack - Start - … - PROFIT It’s still a Postgres. How it works? 63 LIGHTWEIGHT BABY
64. Example 64
65. Example 65 import embeddedpostgres "github.com/fergusstrange/embedded-postgres" logger := &bytes.Buffer{} postgres := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig(). Username("beer"). Password("wine"). Database("gin"). Version(V12). RuntimePath("/tmp"). BinaryRepositoryURL("https://repo.local/central.proxy"). Port(9876). StartTimeout(45 * time.Second). StartParameters(map[string]string{"max_connections": "200"}). Logger(logger)) err := postgres.Start() // do magic err := postgres.Stop()
66. Example 66 import embeddedpostgres "github.com/fergusstrange/embedded-postgres" logger := &bytes.Buffer{} postgres := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig(). Username("beer"). Password("wine"). Database("gin"). Version(V12). RuntimePath("/tmp"). BinaryRepositoryURL("https://repo.local/central.proxy"). Port(9876). StartTimeout(45 * time.Second). StartParameters(map[string]string{"max_connections": "200"}). Logger(logger)) err := postgres.Start() // do magic err := postgres.Stop()
67. Demo 67
68. But… 68
69. - Package is based on Java implementation But… 69
70. - Package is based on Java implementation - https://github.com/zonkyio/embedded-postgres But… 70
71. - Package is based on Java implementation - https://github.com/zonkyio/embedded-postgres - kudos to them But… 71
72. - Package is based on Java implementation - https://github.com/zonkyio/embedded-postgres - kudos to them - That’s why it fetches Postgres wrapped in a JAR But… 72
73. - Package is based on Java implementation - https://github.com/zonkyio/embedded-postgres - kudos to them - That’s why it fetches Postgres wrapped in a JAR - wut? But… 73
74. - Package is based on Java implementation - https://github.com/zonkyio/embedded-postgres - kudos to them - That’s why it fetches Postgres wrapped in a JAR - wut? - Which requires additional xz But… 74
75. - Package is based on Java implementation - https://github.com/zonkyio/embedded-postgres - kudos to them - That’s why it fetches Postgres wrapped in a JAR - wut? - Which requires additional xz - XZ Utils backdoor, anyone? But… 75
76. - Package is based on Java implementation - https://github.com/zonkyio/embedded-postgres - kudos to them - That’s why it fetches Postgres wrapped in a JAR - wut? - Which requires additional xz - XZ Utils backdoor, anyone? - Builder pattern, ☕ But… 76
77. - Package is based on Java implementation - https://github.com/zonkyio/embedded-postgres - kudos to them - That’s why it fetches Postgres wrapped in a JAR - wut? - Which requires additional xz - XZ Utils backdoor, anyone? - Builder pattern, ☕ - personal taste But… 77
78. - Package is based on Java implementation - https://github.com/zonkyio/embedded-postgres - kudos to them - That’s why it fetches Postgres wrapped in a JAR - wut? - Which requires additional xz - XZ Utils backdoor, anyone? - Builder pattern, ☕ - personal taste - … But… 78
79. Can we do better? 79
80. - Of course the answer is “Yes” Can we do better? 80
81. - Of course the answer is “Yes” - Simpler API Can we do better? 81
82. - Of course the answer is “Yes” - Simpler API - Less unpacking work Can we do better? 82
83. - Of course the answer is “Yes” - Simpler API - Less unpacking work - Do we need Internet at all? Can we do better? 83
84. - Of course the answer is “Yes” - Simpler API - Less unpacking work - Do we need Internet at all? - Lovely NIH syndrome :) Can we do better? 84
85. Example 85
86. import "github.com/cristaloleg/embedpg" import "github.com/cristaloleg/embedpg/embedpg18" cfg := embedpg.Config{ Package: embedpg18.Package, Port: 54545, RuntimePath: ".data", DataPath: ".data/pgdata", Username: "postgres", Password: "postgres", Database: "postgres", } instance := embedpg.New(cfg) err := instance.Start() // ... err := instance.Stop() Example 86
87. import "github.com/cristaloleg/embedpg" import "github.com/cristaloleg/embedpg/embedpg18" cfg := embedpg.Config{ Package: embedpg18.Package, Port: 54545, RuntimePath: ".data", DataPath: ".data/pgdata", Username: "postgres", Password: "postgres", Database: "postgres", } instance := embedpg.New(cfg) err := instance.Start() // ... err := instance.Stop() Example 87
88. import "github.com/cristaloleg/embedpg" import "github.com/cristaloleg/embedpg/embedpg18" cfg := embedpg.Config{ Package: embedpg18.Package, Port: 54545, RuntimePath: ".data", DataPath: ".data/pgdata", Username: "postgres", Password: "postgres", Database: "postgres", } instance := embedpg.New(cfg) err := instance.Start() // ... err := instance.Stop() Example 88
89. import "github.com/cristaloleg/embedpg" import "github.com/cristaloleg/embedpg/embedpg18" cfg := embedpg.Config{ Package: embedpg18.Package, Port: 54545, RuntimePath: ".data", DataPath: ".data/pgdata", Username: "postgres", Password: "postgres", Database: "postgres", } instance := embedpg.New(cfg) err := instance.Start() // ... err := instance.Stop() Example package embedpg18 import _ "embed" import "github.com/cristaloleg/embedpg" //go:embed postgresql-18.3.0-blablabla.tar.gz var blob []byte var Package = embedpg.Package{ Version: "18.3.0", Blob: blob, } 89
90. Demo 🤞 90
91. Conclusions 91
92. - Something new is something old Conclusions 92
93. - Something new is something old - Don’t download what is constant Conclusions 93
94. - Something new is something old - Don’t download what is constant - Ah, don’t do crazy 💩 with your infra Conclusions 94
95. Thanks 95
96. Golang Warsaw team 👌 Thanks 96
97. Golang Warsaw team 👌 Nearmap 💕 Thanks 97
98. Golang Warsaw team 👌 Nearmap 💕 You 🫵 Thanks 98
99. Thank you Questions? Email: hey@olegk.dev That’s all folks