Every ship in EVE Online eventually undocks and leaves the station. This time, it’s the whole game. EVE has run on Python 2 since it launched in 2003, all 2.4 million lines of it, on a custom Stackless interpreter that stopped at 3.8 and was archived last year. Destination: Python 3.12. The route runs through 6,500 lines of division that decide who wins a fight, and 100 gigabytes of pickled Python objects that have to survive the jump intact. Kristinn Sigurbergsson was on this show ten years ago. He’s back, with Jamie Bannister, who is flying the EVE Online migration right now, and Thomas Dähling, who took EVE Frontier through first. In EVE, a destroyed ship is gone for good. There are no do-overs, and the universe has to stay online the whole way.
00:00 Every ship in EVE Online eventually undocks and leaves the station.
00:04 This time, it’s the whole game.
00:06 EVE is run on Python 2 since its launch in 2003.
00:10 All 2.4 million lines of it on a custom stackless interpreter that stopped at Python 3.8.
00:18 And stackless Python was mothballed last year.
00:21 Now it’s destination Python 3 for EVE Online.
00:24 The route runs through 6,500 lines of division that decide who wins a fight
00:29 and 100 gigabytes of pickled Python objects that have to survive the jump intact.
00:34 Kristinn Sigurbergsen was on the show 10 years ago, and he’s back, this time with Jamie Bannister,
00:39 who is flying the EVE Online migration right now, and Thomas Darling, who took EVE Frontier through the jump first.
00:47 In EVE, a destroyed ship is gone for good.
00:49 There are no do-overs, and the universe has to stay online the entire time.
00:54 That’s a serious migration.
00:56 This is Talk Python To Me, episode 564, recorded Thursday, September 10th, 2026.
01:20 Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists.
01:24 This is your host, Michael Kennedy.
01:26 I’m a PSF fellow who’s been coding for over 25 years.
01:31 Let’s connect on social media.
01:32 You’ll find me and Talk Python on Mastodon, BlueSky, and X.
01:35 The social links are all in your show notes.
01:38 You can find over 10 years of past episodes at talkpython.fm.
01:42 And if you want to be part of the show, you can join our recording live streams.
01:45 That’s right.
01:46 We live stream the raw uncut version of each episode on YouTube.
01:50 Just visit talkpython.fm/youtube to see the schedule of upcoming events.
01:54 Be sure to subscribe there and press the bell so you’ll get notified anytime we’re recording.
01:59 This episode is sponsored by Sentry’s Seer.
02:02 If you’re tired of debugging in the dark, give Seer a try.
02:05 There are plenty of AI tools that help you write code, but Sentry’s Seer is built to
02:09 help you fix it when it breaks.
02:11 Visit talkpython.fm/sentry and use the code talkpython26, all one word, no spaces,
02:17 for $100 in Sentry credits.
02:20 And it’s also brought to you by Talk Python Courses.
02:24 Course completion certificates are now live.
02:26 If you finished a course, there’s a certificate waiting for you on your account page right now.
02:31 Download it as a PDF or add it to your LinkedIn profile with one click under licenses and certifications.
02:37 Same section as your formal degrees.
02:39 Visit training.Talk Python.fm slash account to see what you’ve already earned.
02:45 Kristinn, Jamie, and Thomas, welcome to Talk Python To Me.
02:49 Kristinn, welcome back.
02:51 Jamie Thomas, hello.
02:52 Welcome to the show.
02:53 Hello.
02:53 Hello, Michael.
02:54 Hello.
02:55 Great to be back.
02:56 Yeah, yeah.
02:56 Great to have you all here.
02:57 So I’m really excited to be talking about EVE Online again.
03:02 When was it we?
03:03 I think it was 2016, 17.
03:06 It was a while ago.
03:07 I think it was 2016.
03:10 And I think we did even brought up the discussion point of the Python 3 upgrade.
03:16 So it’s been on the table for a while.
03:18 Well, we’re here to talk about upgrading EVE Online to Python 3.
03:22 Well, Virgin, are you going to start at 3.0 and work your way through all of them,
03:26 or are you going to do a little bit of a skip ahead?
03:29 So we are going to Python 3.12.
03:33 That’s what we updated to.
03:34 We first did a step in between to stackless Python 3.8.
03:38 But when we did the work, 3.8 was already end of life, so to speak, and stackless Python didn’t go any further.
03:45 So we decided to go to 3.12 and have now worked on our infrastructure pipelines a little bit that any upcoming updates are a lot easier.
03:53 So we have plans to go to 3.15 later this year, for example.
03:57 Oh, fantastic. That’s really neat. I’m going to be excited to hear about that.
04:00 Probably some cool performance benefits there and profiling and other types of benefits that you get from those last few versions there.
04:08 We’re going to dive into why stackless and all that.
04:12 There’s a lot of history here, and it’s a really, really big project.
04:15 Before we get to the project itself and the upgrade and so on, let’s just start with a quick introduction for each of you.
04:24 I’ll let you do that for yourselves.
04:25 Kristinn, you go first.
04:26 We’ll go around the squares on the screen here.
04:30 Okay.
04:30 My name is Kristinn Seupersson.
04:32 I’m a director of gameplay engineering for eFrontier, which shares the same technology stack as eOnline.
04:41 I’ve previously been everything from a game designer to software engineer to technical director of E1 Line and later technical director of platform or SEAR technology segment as well.
04:51 So I was part of the team that did the migration for 2.5.3 for Frontier and now I’m the primary benefit from it.
05:03 So I’m Jamie. I’ve been an engineer on EVE for about 16 years, mostly on gameplay.
05:11 But that has let me work across almost all the different aspects of the game,
05:15 from the front end of the client all the way through to the back.
05:18 So I’ve kind of seen all of the guts and the glory inside EVE.
05:23 So I’m at the moment working on the actual migration project for EVE Online itself,
05:28 building on kind of what the others have done previously.
05:31 Excellent. Thomas?
05:33 Yeah, I’m Thomas Delling. I’m a principal programmer on the platform segment for Fenris
05:39 Creations. And I have been leading the initial Python 3 transition for Carbon, like getting
05:45 us out of 2.7 onto Python 3.
05:48 Amazing. What a big project, y’all. What a big project. I think where we should start
05:53 is just what is EVE Online? And I have, of course, EVE Frontier pulled up. But tell us,
06:00 what is EVE Online? Whoever wants to take this one.
06:03 So Eve Online, it’s an MMORPG in the classic sense.
06:09 It’s been going since 2003.
06:12 So it’s a massively multiplayer online game set in the far future where humanity has gone through the wormhole to a new system
06:21 and society has kind of collapsed and then rebuilt in a new form.
06:25 It’s a very, what we would call in gaming, an open world sandbox.
06:30 So we support all sorts of gameplay professions from mining, trading, piracy, a lot of combat.
06:38 I think the distinctive factor is the game is full, kind of full loss.
06:44 If you kill someone’s ship, that’s a ship removed from the game.
06:47 There’s no kind of getting things back.
06:50 Nothing lasts forever.
06:51 So the game is built on creation and destruction.
06:54 A big part of it, if I understand it right, is kind of the economy and just the society.
06:59 Like there’s a society that forms from the players.
07:03 Yeah, the economy I think, and before I joined to work on the game, I was a big player as well,
07:11 and the economy was a big part of it for me.
07:13 It’s been used by, or been studied by economists, because it’s, if you think,
07:19 the data that we have on a big, real, on our economy is more accurate and detailed
07:24 than you can kind of get from studying the real world economies, where you’ve always got imprecise data.
07:29 So we have every single transaction.
07:31 We know everything that’s going on.
07:33 You see all the money as it moves around the game.
07:36 That’s super interesting.
07:37 Yeah, very, very rich simulation.
07:39 What if we lived in a society where there was zero privacy and 100% data sharing?
07:44 It would be creepy from a human perspective, right?
07:47 Just like governments and alt-territarism and all that kind of stuff.
07:52 But from an economics perspective, you have perfect data, right?
07:56 Yes, yeah.
07:57 and it’s interesting how real world things have come up in the game like there’s no formal mechanic
08:03 for banks for example but players themselves have built banks they’ve built uh kind of loan schemes
08:10 organizations have got their own way of paying their members for turning up to operations so they
08:19 they have all the equivalent of like we have welfare groups we have um very centralized
08:24 democratic groups, communist groups, all those kind of different social phenomena have reappeared
08:30 in the game kind of organically. So interesting. And many online games are, you go here, you find a
08:38 server, you join 40 people, and you go have an experience, right? But this is a single, unified,
08:44 one world that everybody is in. They can all interact, all that kind of stuff, right? Yeah.
08:49 So we take place where we call it a single shard.
08:52 So yeah, like you say, it’s one universe.
08:55 Any two people in the game, if they go to the same place, they will meet and see each other.
09:00 And so we have, I think our largest kind of player group is about 15,000 characters.
09:10 And so all of those potentially could get together and go to one place at one time and have a huge fight.
09:15 I think we have two Guinness World Records for the most, uh people taking part in a pvp fight wow how do you coordinate that as a as a group you know what
09:26 i mean like how do you coordinate just the conversations amongst the people without it
09:32 just being a cacophony um between in turn ourselves i mean with the players i mean with the players
09:38 if there’s 1500 people in like uh some sort of action how can you have conversations and
09:44 communication without it being they built built their own solutions for a lot of this um so they’ve
09:51 replicated a lot of a chain of command that you get from from real kind of real world and some of
09:55 the bigger groups they have their own it departments uh that take care of of things a lot of groups use
10:02 uh discord for kind of coordination but they have and various other kind of voice uh systems they
10:08 have uh what’s called an fc which is like a fleet commander so that’s the person uh giving the orders
10:14 and kind of directing the troops.
10:16 And then it kind of, there’s a hierarchy, if you like, of communication that goes down.
10:21 So yeah, the bigger groups have built very complex kind of command and control systems
10:28 with themselves.
10:29 I guess that makes sense, but wow.
10:31 And you think of a real army, if there’s a thousand people, they don’t all just get onto the same audio channel.
10:37 They do have certain people who talk to each other and then they give commands down the line.
10:42 And interesting.
10:42 They replicate the processes like scouts filtering information up.
10:46 They’ll have then some people gather that information and feed it into the FC,
10:50 who makes the calls of where they’re going to go, what they kind of do next.
10:54 Yeah.
10:54 Kristinn, Thomas, you guys want to add anything to that overview before we dive in?
11:00 Yeah, I would say that this doesn’t really happen by its own.
11:03 There are a lot of people that play EVE online that spend an incredible amount of time orchestrating all this.
11:11 not only just the fights themselves, but the reasons behind the fight, gathering everybody up.
11:17 So it is really fascinating to see. And to some extent, these people are playing the game,
11:22 but they’re not actually in the game because they are managing the spreadsheets and
11:28 the communications and all of that. And to some extent in certain circles, like people have
11:34 put their experience in managing these larger corporations in EVE onto their CV and it has
11:42 helped them get a job. That’s a level of commitment there. Yeah, I can now function as a full
11:50 logistics operation person because I’ve done it for a thousand people in EVE Online.
11:56 Very much so. Yeah, very much. Crazy. Thomas? Oh, I’ve got nothing more to add there. I think
12:01 they both described it really well.
12:03 It can sometimes be almost more like a job than a game if you want it to be.
12:07 You learn a lot of valuable things.
12:10 You meet a lot of great people that way as well, right?
12:12 I mean, over the 20 years, so many people that met each other.
12:15 We even had a wedding at one of our FanFests, I believe.
12:20 We’ve actually had a few, I think.
12:22 Thomas, you gave a great talk at FanFest about upgrading and inspiring the folks.
12:28 That video is online.
12:30 I’ll link to it in the show notes.
12:31 We’ll maybe get to that in a little bit later.
12:33 So that sounds like a really cool gathering.
12:36 You know, I really, the last couple of years, I’ve really enjoyed just diving into games.
12:41 And my daughter is getting older.
12:42 And one of the things she really likes to do is sit down and play games together.
12:46 Not even online, it’s too big, but sort of cooperative stuff and just sort of exploring.
12:51 And I think that there’s, well, I know that studies have shown that there’s more loneliness.
12:57 There’s fewer third places that people find themselves at and all that.
13:02 Having something like this where you can gather and be with people and immerse yourself in,
13:07 I think it’s really neat, honestly.
13:08 As long as you don’t take it too far, you know, still go to work and stuff.
13:11 Yeah, yeah.
13:12 I mean, before I joined and as a player, we would organize meetup events at different places around the UK where I was from.
13:22 And so I got to know a lot of people.
13:24 I travel to a lot of places just for the purpose of meeting up with other EVE players.
13:29 So for a lot of people, EVE or a lot of games like this, they can be your primary social connection to a lot of them.
13:38 Let me take us back 35 years or something like that to the days of BBSs.
13:46 Have any of you guys done BBSs? Remember them?
13:49 Yeah, I used to play the MUDs back in the day.
13:51 Oh, I used to play MUDs as well.
13:53 Though that’s not what I’m thinking of.
13:54 But actually, MUDs have a really interesting multi-user dungeons.
13:58 They have some really interesting analogies as well.
14:00 But I was thinking of Trade Wars.
14:03 Have any of y’all played Trade Wars from the BBS days?
14:06 I know of it.
14:07 I’ve never played it.
14:09 It’s really interesting.
14:10 It was this multiplayer thing way back in the 90s.
14:13 And you would dial in and you’d have to take turns because only one phone could be connected to the BBS at the time, at a time.
14:21 And it was really, it has some real similar vibes.
14:24 It’s, you know, you travel to different planets.
14:26 There’s mining, there’s an economy, there’s trade.
14:28 Anyway, I feel like this is like, what if it could be real?
14:32 Instead of just a little sort of simple game, what if you could make it real with all the nuances
14:37 and intricacies of reality and multiplayer?
14:39 And it’s just the realization of what that thing should have become.
14:43 Yeah.
14:45 Because Eve was heavily inspired by a game called Elite, which was one of the very early 3D space games.
14:52 And that had a big world, a big universe of systems.
14:57 Trading was a big part of it.
14:58 And so there’s definitely links from there.
15:01 And also classic role-playing games as well.
15:04 Our character stats have similar kind of perception, intelligence, charisma sort of stats.
15:09 Interesting.
15:10 Almost a D&D sort of.
15:12 Very much, yeah.
15:13 Yeah, you can tell there were D&D players in the very early EVE designers, for sure.
15:18 This portion of Talk Python To Me is brought to you by Sentry and Sear AI.
15:23 There are plenty of AI tools that help you write code, but Sentry’s Seer is built to help you fix it when it breaks.
15:29 The difference is context.
15:31 Sear isn’t just guessing based on syntax.
15:33 It’s analyzing your actual Sentry data, your stack traces, logs, and failure patterns.
15:38 Because it has the full context, it can A, spot buggy code in review and help prevent issues before they happen,
15:45 and B, identify the root cause of production errors.
15:49 It can even draft a fix and hand the work off to an agent-like cursor to open a PR for you.
15:54 Seer turns Sentry into a complete loop.
15:56 You have your traces, errors, logs, and replays to see the problem, and now AI to help solve it.
16:01 Join millions of devs at companies like Claude, Disney+, and even Talk Python, who use Sentry to move faster.
16:08 Check them out at talkpython.fm/sentry and use code talkpython26, all one word, for $100 in Sentry credits.
16:17 Thank you to Sentry for supporting Talk Python.
16:20 We’re going to also talk about this thing called Eve Frontier, which is a little different,
16:26 but very much inspired, shares a lot of the code base.
16:28 So give us a quick comparison to Eve Frontier, which I’m sure fewer people know about.
16:33 Yeah, sure.
16:34 So what matters for the tech is that we’ve essentially forked Eve Online, the code base,
16:41 and are sharing a lot of the underlying tech.
16:44 We’ve made a lot of modifications.
16:45 We haven’t been kind of pulling in stuff lately because it’s just, it’s become too different.
16:53 Where it differs from EVE Online is maybe if you play EVE Online, you very much come into a society.
17:00 Part of it is definitely player built, but part of it is also kind of NBC built.
17:05 The environment is there’s a police, there’s everything.
17:08 Whereas in Frontier, you’re more coming into like an unbuilt, uninhabited zone
17:12 and you’re the first kind of writers in that universe.
17:15 And it’s up to the players to build that society up and kind of build up all of that infrastructure.
17:23 It doesn’t really differ from even a lot of like the design paradigms.
17:29 I think both games aim to give players as much player agency as they possibly can.
17:34 But this also allows us to make like a lot more dramatic changes in the gameplay itself.
17:42 it’s very hard to change the gameplay on evil line for example very drastically because honestly
17:47 like after 20 years that’s the game that people are playing they don’t really like they didn’t
17:51 play it for 20 years to change all of it so uh we’ve been doing that it’s more survival
17:58 survival based it’s more like uh like a survival game than it is uh uh not necessarily more than
18:05 an mmo but it definitely has a lot more survival elements to it uh yeah and the environments are
18:10 a little bit more kind of claustrophobic and smaller.
18:13 Obviously, that’s very hard in a space game because space is pretty vast.
18:17 Yeah, amazing.
18:18 Okay, really, really neat.
18:20 Now let’s talk about the reason that I reached out to you, which is I saw this article or post from you all
18:27 that said the move to Python 3 begins.
18:29 And, you know, credit to you all.
18:31 This is a really fun article that it’s like, it’s told as sort of the next phase of a game
18:37 or something where it’s kind of a mission in the game, I guess.
18:42 But a mission for you all, right?
18:44 Yeah, I think that we kind of got a little bit cheeky with the heading because it isn’t necessarily the beginning
18:51 because, like we were saying earlier, Frontier itself is already on Python 3.
18:57 But this is specifically for EVE Online itself.
19:00 It’s now actively moving to Python 3.
19:02 And a lot of it is building up on the tech that we’ve developed ready and kind of start to prove out through Frontier. But yeah, this is kind of our big
19:12 formal announcement that the legacy of 20 plus years of code, we’re now kind of actively working
19:20 to migrate. I think we should start maybe with a little bit of the architecture and just a big
19:26 tech picture of EVE Online. When we spoke, Kristinn, way back in the day, you were doing a bunch of
19:32 interesting things. I think you must have been the single largest deployment of Python,
19:37 a Python app on Windows?
19:39 Is that correct?
19:40 I would say yes, right?
19:41 What else would it be?
19:42 That seems very plausible.
19:44 I think it was a very kind of like a niche environment.
19:48 And what I’d often run into is that I would try to take some libraries and try to use them and they just wouldn’t work on Windows.
19:54 Like somebody broke it and nobody cared because they weren’t really using it on Windows,
19:59 except we were.
20:01 Obviously, like at that time, like 10 years ago, we couldn’t really use a lot of, especially native Python
20:07 libraries.
20:08 I can’t even remember if we had removed-
20:10 we must have removed.
20:11 We had our own importer as well, which made it also difficult to use third party libraries.
20:17 But I think we fixed that.
20:20 Yeah, there’s a lot of-
20:21 I mean, it’s a very old project.
20:23 And a lot of the early beginnings in the code base were before a lot of the things in Python
20:27 actually being standardized to a degree.
20:30 plus the special circumstances of having a packaged program with its own Python interpreter
20:37 just makes certain things more interesting.
20:39 Like, you know, as Kristen alluded to, like you can’t just import a random public package
20:44 because we don’t have pip as an example, right?
20:47 Or uv these days.
20:48 But like, there’s a lot of interesting problems you need to solve when you have such a large deployment,
20:53 for sure.
20:53 Yeah, there was a lot of custom stuff.
20:55 When did this first get created?
20:56 So I know we spoke in 2016, but it’s been around for more like 20 years, right?
21:00 So 2005, 2006?
21:04 I think it’s earlier than that.
21:06 It’s like the game is released that 2003, the development started in 99, I think.
21:15 They didn’t really choose Python.
21:18 I don’t know the exact date of that, but like there was, that is an interesting story as well.
21:25 Like why they chose Python, which was actually rather new at the time.
21:29 Yeah, I was actually wondering about that as well.
21:32 I’m glad you chose Python.
21:33 I think it’s really cool.
21:34 And I’m sure you all really enjoy working on it.
21:36 And this has got to be an exciting project to say.
21:39 We’re moving to 3.15.
21:41 We can use uv.
21:42 We can import libraries, like all the exciting stuff.
21:46 But at the time, it was pretty new.
21:48 And we said, well, we could run this much faster and easier if we wrote it,
21:52 I don’t know, with C++, Java, C#, something like that, right?
21:57 I think a lot of the core libraries, they are in C++, so our physics engine, which we call Destiny, that’s all C++. Trinity, our graphics engine, similarly. But yeah, most of our kind of higher level, all the gameplay logic, a lot of the network stack that we have, that’s all built, or clustering stack, that’s all built in Python.
22:25 Yeah, probably add to that as well is that like the time when we started or not me, even though I’ve been here for a very long time, it hasn’t been that long.
22:37 But when they started writing EVE, like that was around the.com era and kind of the older listeners probably remember that era.
22:45 But the programming resources were scarce and heavily sought after.
22:49 So very expensive.
22:51 So part of the reason why they chose Python was to get more junior people and even designers
22:57 to be able to write sequential code in Python in order to speed up development.
23:04 It was all about development speed.
23:06 Especially in Iceland, right, where you’ve got a much smaller pool of people to work from.
23:12 Honestly, that’s kind of a cool success story, right?
23:15 Like bring more people in to help build the thing that we’re building.
23:19 Yeah, absolutely.
23:21 Yeah, absolutely. And as Kristin just said, allowing people to write sequential, simple, straightforward Python code as well, which then led to us discovering stackless Python, integrating that one, and then building an abstraction on top that people don’t realize that they are writing code that runs asynchronously.
23:40 And it enabled us to scale to the degree that we’re able to scale with thousands of people being in the same solar system and so on.
23:47 Yeah, you needed concurrency before Python was inspired by concurrency.
23:52 Maybe we should talk a little bit about free-threaded Python later, as well as the possibilities there.
23:58 But I just want to kind of touch on this historical angle a little bit more.
24:02 To me, there’s two really significant milestones that matured Python.
24:07 I’m sure there are more, but these are the ones that stand out in my mind.
24:10 They both are around 2006 or so time frame.
24:14 So one is Django coming to Python, really making it a first class way to build web apps.
24:21 People start building on top of that.
24:23 That puts a lot of pressure to make Python operate a little bit differently, right?
24:28 10 years later, we’ve got Instagram running the largest Django deployment in the world
24:32 and doing things like trying to juggle weird memory stuff by turning off the GC, for example,
24:39 entirely and other things along those lines.
24:41 The other one around 2006 as well is the invention of NumPy and then later Jupyter and the whole data science stack, which also put really hard computational pressure on Python rather than, say, the parallelism of web apps and so on.
24:56 You all were before both of those, right?
24:58 So it was early, early days.
25:00 Yeah.
25:01 I mean, what we also found was, or what I found in some of work here is the original developers had to invent a lot of things which didn’t exist.
25:11 So like the containerized system, a way to kind of cluster an application across multiple
25:19 CPUs and notes.
25:21 There wasn’t really a lot of easy off the shelf frameworks for that kind of thing.
25:24 So they had to roll their own.
25:27 Even logging and telemetry systems, there weren’t great ones of those.
25:32 So what we found from the early years is we have a lot of in-house tools, which we made
25:39 or the people at the time made, which are now using the models that you see being replicated
25:48 by much more popular systems.
25:50 So whether it be Redis for a storage system or your containers as well.
25:56 The way that maybe we can talk about this a bit more in a bit is the way we distribute
26:00 EVE as an application across about 200 nodes in our server cluster.
26:07 And each of those has certain jobs to do.
26:11 And so we have to deploy and configure all of those as one giant cluster.
26:16 And we have our own coordination tools that have been built over the years.
26:20 Nowadays, if you were starting this from scratch, you would have so many off-the-shelf solutions that we just didn’t have the options for.
26:26 Oh, you just set up Kubernetes and then do this auto-scaling thing.
26:30 And it’s, oh, yeah, it’s just easy.
26:33 I would also like to add, I feel like the data scientists kind of grabbing onto Python and
26:40 I feel like pushing it towards Python 3 because I think without that army going into Python
26:48 3, I feel like it mirrors the rationale of why we chose to go into stackless Python.
26:53 You were giving basically a tool to a bunch of people that aren’t really programmers,
26:57 but they need to write code.
26:59 So I feel like that is an interesting thing.
27:03 Yeah, you know, I’ve forgotten the 2v3 wars and all that.
27:07 Now, yeah, it’s easy to forget the details of that from, it’s been a while now, actually,
27:12 but you’re right.
27:13 It was the data scientists that were going into Python 3, and the rest of the folks were
27:18 kind of dragging their feet because they had existing code bases that needed to keep running
27:23 and keep being migrated, whereas the data science projects are more frequently like,
27:27 we’ll start a new project to answer this question or to study this new bit of data.
27:32 So they had many more opportunities to choose the most modern tools.
27:37 And I think, honestly, we should give the data scientists a little bit more credit of making the two to three change happen.
27:44 All right, well, let’s talk stackless Python because this is one of the foundations
27:49 that you all had to migrate from.
27:51 And the journey begins, The move to Python 3 begins article talks about upgrading from 2.5 to 2.7 not too long ago.
28:02 And then finally up to 3.8.
28:06 What is it about Stackless Python?
28:07 I know it has strong origins with you all because some of the core developers worked on it and refined it there.
28:14 But this far down the line, I imagine there’s a lot of people that don’t know what this is.
28:19 Yeah, absolutely.
28:21 Stackless Python itself was a very great solution at the time, right?
28:25 I mean, it’s basically Golang’s GoRoutines, but in Python, before things like this co-routine model
28:32 became so popular as it is in recent years.
28:35 A big problem that we faced was, on the one hand, that because of the customizations
28:42 we had to do to the network stack to make all the magic of people writing sequential code, but it does end up not blocking
28:49 the main process.
28:51 That was a big learning curve for a lot of people.
28:54 And also always something that even the experienced developers tended to forget about in a moment.
29:00 And they’re like, oh, yeah, hang on, there’s a DB call.
29:02 Of course, that we yield.
29:03 I need to put a lock here or whatever.
29:06 So the model in itself is great, but it comes with a few learning hurdles, I want to say.
29:15 And it served us really well.
29:17 One problem that we had over the years that had become more and more interesting, especially
29:22 last 15 years, I want to say, is that we relied on a heavily customized version of it. So stackless
29:28 Python itself is a fork of the CPython interpreter with a lot of changes in the interpreter. And
29:35 we put some changes on top because doing a real-time game of sorts requires some solutions that go
29:43 very far down the tech stack to be performant. Like if you want to get certain metrics, if you want
29:47 to control how memory is allocated and so on and so forth right so all these customizations and
29:53 also meant that made it very difficult for us to like always go to the next newer version of python
29:58 as the article current calls out like you know there was some code that was still python 2.5
30:02 when we did the native extension porting to python 3 we discovered things from python 2.3 that we were
30:07 still using that shouldn’t have been used but we somehow kept around on our customized version of
30:12 of Stackless Python.
30:14 This portion of Talk Python is brought to you by Talk Python courses.
30:18 Here’s the thing that always bug me.
30:19 You finish one of our courses, that’s hours of video, a pile of code you actually wrote,
30:24 and real skills you didn’t have a month before, and then nothing happens.
30:28 No paper, no credential, nothing to show for it.
30:31 So we fixed it.
30:32 Every Talk Python course now generates a completion certificate automatically.
30:36 Go to your account page in your dashboard section, scroll down to your completed courses and click certificate.
30:42 That’s the whole process.
30:44 Two things you can do with these course completion certificates, download the full PDF,
30:48 which is handy if your employer reimburses training or gives you credit for finishing it.
30:53 Or you can make the certificate public and hit share on LinkedIn, which adds it to your LinkedIn profile under licenses and certifications.
31:01 Not a poster that scrolls away in a day, an actual credential sitting on your profile
31:05 where your manager and recruiters can see it.
31:07 Plus, if you’ve been taking our courses for a while, you’ve probably earned several of these without even knowing they existed.
31:13 Just visit training.Talk Python.fm slash account and collect them.
31:18 Thanks to all of you who have taken a Talk Python course.
31:21 It’s a great way to support the podcast.
31:24 Another aspect certainly is that stack lists had become like a sort of niche project in the ecosystem overall.
31:32 With Python 3, we have async.io built on the twisted model.
31:37 And even in the Python 2 world, G event had become a lot more famous than Stackless Python,
31:43 presumably because you could just import G event in a standard CPython interpreter,
31:47 while Stackless Python is more complicated to get started with.
31:53 So-
31:53 So Stackless Python is basically its own runtime.
31:56 Yeah, it’s basically-
31:57 Yeah.
31:57 Yeah, very much that.
31:58 It’s like its own Python runtime.
32:00 You kind of need your own.
32:01 Like, it has a stackless lib, so to speak, which basically monkey patches similar to what
32:06 Gvent does all of the threading and socket modules and so on to be able to do them in non-blocking fashion.
32:12 And yeah, all these things.
32:14 So that’s really, I mean, that’s what made it possible.
32:17 That’s really awesome.
32:18 One thing, just, I don’t know if you all know this, but how is it possible that it says it’s forked from CPython on GitHub
32:25 when CPython itself didn’t go to GitHub until like the 20s, you know, 2020, I don’t know what it was,
32:31 but maybe some GitHub magic, I don’t know.
32:34 Oh, it’s git magic.
32:35 You will find on GitHub there are repositories where there are commits from Abraham Lincoln and various other historical figures because you can just make up all the history, right?
32:45 Interesting.
32:45 Yeah, it is nice, though, actually, that it shows it.
32:48 If you go to the repo now, it says that it was archived in 2025, which, you know, not every project is meant to last forever.
32:57 But it made it possible for you.
32:58 Although I’m sure that you would think, you know, all right.
33:02 You’re like, wouldn’t it be cool if we could use, I don’t know, libraries like Pydantic or FastAPI or AIO Redis or I don’t know, whatever, right?
33:11 Anything that is new and fancy.
33:13 And then you go back and you look and you’re like, well, but not here.
33:17 Not on Stackless, right?
33:18 Like that was one of the major problems, wasn’t it?
33:21 Yes, this was a major headache for us.
33:24 But for every native extension-
33:26 or not just native extensions– for every Python package we took on, we had to take a really close look
33:32 about how can we integrate it in our stack.
33:34 Again, we didn’t have a package manager.
33:36 We still don’t have it, but we’re working on it.
33:39 And that just meant that you would take the zip file from PyPy, for example, extract it into our environment,
33:45 and then see what happens.
33:47 If that then happened to be something that had a native extension, well, good luck.
33:51 Does this support our minimum spec environment?
33:53 does this run on all our environments?
33:55 Does this work with our version of the forked Python interpreter?
34:00 And we have discovered a few cases where we took on modules that were built for 2.7.18,
34:08 and they then corrupted memory because in 2.7.6, the memory layout of the PyType object changed,
34:14 and we didn’t have that change.
34:16 And yeah, things like that.
34:17 So we had to fight the way forward.
34:20 Jamie, you want to jump in?
34:21 Yeah, no, I was just going to say when it comes to support as well, because we support Mac and we have in the past,
34:28 but we don’t now support Linux as well.
34:30 So sometimes with a native extension, we’ve got to figure out does it even support all of our clients?
34:38 Our servers run on a Windows cluster.
34:42 Most of our players are running on Windows clients.
34:45 Mac is also kind of a smaller majority.
34:48 And like I say, we used to also support Linux.
34:51 So keeping all of those in sync as well with all of these extensions has been problematic at times.
34:57 Yeah.
34:58 Sure, that’s an understatement.
34:59 Earlier, when you guys mentioned how you deploy as an app, or.app,.exe, whatever, or to your server clusters, how do you do that?
35:09 That’s honestly one of the biggest challenges of Python that many attempts have been made, but I honestly don’t think that there’s a good answer, a really foolproof answer yet.
35:18 And until it’s built into CPython itself with a –build flag or something, I think it’s going to be 90, 95% solutions.
35:28 But you all have something that works.
35:29 What are you doing?
35:31 So effectively, I mean, with Python 3, this all got a lot easier because you can initialize the interpreter a lot more straightforward.
35:38 But what we have been doing for the last 20 years is effectively we are embedding the Python interpreter in a custom binary that we have, a custom library that we use.
35:48 And from that moment onwards, we basically just heavily customize what paths are available to the interpreter
35:55 and what environment variables are considered, all these little things.
36:00 And then we package up the remaining things that we need in a way that it fits this layout that we set up
36:05 in the embedded interpreter.
36:07 But yeah, it’s also more like a 75% solution than a real one because the Python build process in itself
36:14 requires some system-specific paths.
36:17 We had to go to great lengths to patch that out.
36:20 Not necessarily in the best way possible either, but it gets us there.
36:24 If you’re shipping it, that’s off you all.
36:27 That’s pretty impressive.
36:28 It was kind of like that we have a solution for our players, but also we built a lot of tools for game designers
36:35 to use and stuff like that.
36:36 And it’s convenient to write them in Python because we can share some of the code base of the actual game,
36:41 game logic, for example, when we’re creating static data.
36:44 And that has been like a very painful experience of distributing Python,
36:50 like Python, basically a Python script to developers, because there is really amazing how much people can mess with their environment
36:59 by just changing the Python path, running the wrong version of Python.
37:04 It’s been a struggle for years, honestly.
37:07 Even if we get a new hire, the first few days are often spent setting the machines up.
37:13 And inevitably, as Windows slightly changes from release to release, it will or won’t come with certain things.
37:21 Sometimes we have to disable the built-in Python that ships with Windows just so that we can use our own specific Python 2.7 versus Python 3.
37:31 So that’s also quite a lot of fun that we have to get started out.
37:36 What about things like uv Run or uvx in the future?
37:41 There’s some interesting possibilities there, potentially.
37:44 Absolutely.
37:45 And as I said, we are working on it.
37:47 We are exploring the angle there.
37:50 Because it is still a bit of a hassle of getting things into our special environment for the game itself.
37:58 We have historically had also the issue that we were using a different built environment
38:02 than the standard Python distributions, right?
38:05 Partially because there’s a lot of code that would need to be migrated.
38:08 Partially because it works.
38:11 We don’t want to spend resources on that at the moment, partially because, well, especially with Stackless Python,
38:18 you could only go so far until there’s a new compiler version that then breaks some internals, and you would have
38:22 to do major rewrites.
38:24 So these issues have existed, but we want to be able to be in a place where a developer can also
38:31 just say, “uv install this package into my game environment,” then be able to use it.
38:37 Needs to solve a few things, like if it’s a native extension, needs to find the right build environment and so on, which is non-trivial, unfortunately.
38:45 What workflows do you think, do you see becoming possible that were much, much harder once you
38:50 have nice package managers and maybe more reuse, right? For example, you could ship versions of
38:57 things more easily than, you know, as one monolithic element. Yeah, for sure. I mean, for example,
39:03 we recently open-sourced basically the game engine, right? And there is, I can see a world where you
39:09 can just uv install or pip install these things into your custom interpreter to get going or into
39:15 your Python interpreter to get going without requiring our custom interpreter. For our internal
39:21 workflows, I don’t think, I mean, sure, the acquisition of an external package will definitely
39:27 get a lot easier, but many of the other workflows probably won’t change that much since we,
39:33 I was in 2011 or 12 or so, we added an interpreter mode to the game engine itself, so you can
39:38 can basically start the game engine with a special flag, and then it basically starts
39:42 the Python REPL, and you get as much of the standard Python interpreter behavior as possible.
39:49 That’s cool. So you can kind of poke around that in the game memory and values and data
39:54 structures as you want, right?
39:56 I guess the director in me doesn’t really, it doesn’t see it as all negative that there’s
40:03 a bit of a hassle to add a new dependency.
40:07 I was thinking about that too, Kristinn. I was. And you can see these headlines of all the supply chain issues and you’re like, well, I mean, we should worry about it, but not to the degree that other people with 200 dependencies do, right?
40:19 Yeah, definitely. And I’m not arguing necessarily for making it hard, but you can see a reality when you have a large development team and people just kind of flippantly add the dependency that causes issue. It’s very vulnerable to individual mistakes.
40:38 Yeah, and also just duplication of now there’s three libraries that kind of handle the same thing, but not entirely.
40:45 On the flip side, if it’s harder to upgrade, you’re also less likely to actually upgrade it out of the.
40:53 What libraries or tools or experiences did you all see out there and you’re like,
40:58 all right, we have to upgrade.
41:00 Were there certain packages you’re like, oh gosh, we really, really would benefit
41:04 from using this one or having this feature like the JIT or something in Python
41:10 that makes such a difference that it’s worth all this effort.
41:13 So, I mean, Jamie, maybe you can talk about the Python side a little bit,
41:16 but from the game engine perspective, the main driver for really wanting
41:22 to get the upgrade project pushed through was the native backline that we worked on in the early 2020s.
41:31 It has just like Python 2.7 was not a way of silicon to begin with.
41:36 Then a lot of other changes that had happened, like macOS behaves very different with Unicode
41:42 than Windows does when it comes to system calls and so on.
41:45 Historically, before we had the native MacWords, clientists would go through a translation layer similar to Wine for Linux.
41:52 So we really pushed for it because we just saw that, okay, if we don’t go up to Python 3,
41:58 we are stuck with having a small team that is dedicated to nothing else but maintaining the Python interpreter.
42:07 That seemed like it’s one possible solution, but not necessarily the best one if you want to just go
42:14 rapidly than to add new features or iterate on other existing functionality.
42:19 I think for me as a gameplay programmer who’s mostly using the Python framework that we have,
42:26 things that we’ve sorely been lacking for a long time is a good way of type declaration.
42:31 It’s sort of there in Python 2.7 if you’re using the right IDE, but it’s not consistent.
42:38 That gets a lot better as you move through the Python 3 versions.
42:41 Another thing we end up making a lot of is what’s effectively a data class,
42:46 where we just want to kind of define a small packet of data in a particular structure
42:50 and pass it around and do things.
42:52 Having kind of the language support that at a more fundamental level is kind of something I’m really looking forward to getting.
42:59 Yeah, that’s really neat.
43:00 And think about such a large code base.
43:02 Was it 2.4 million lines of code, something like that?
43:06 That’s non-trivial.
43:06 I mean, that’s kind of the scenario that typed Python is perfect for and could have been invented for before AI was a thing,
43:14 which also helps.
43:15 But having the ability to use tools like Pyrefly or ty and ask the question in an automated way,
43:21 is our code still consistent?
43:23 That’s pretty neat, right?
43:24 Yeah, I think given the size of our code base and the fact that we have reasonable test coverage,
43:32 but we don’t have complete by any means.
43:34 So the more we can put in a kind of leverage from the language to help prove correctness and find issues,
43:41 that’s going to lead to a better product and less issues.
43:44 Yeah.
43:44 Last question before we actually get into the details of the migration and how you all did it.
43:49 What about free-threaded Python?
43:51 You started with this concept of high concurrence, so much so we need concurrency that we’re going to get our own runtime and maintain it.
43:59 Are you considering free-threaded Python?
44:01 Is this interesting in any way?
44:03 Or is it all asyncio driven?
44:06 What are you considering here?
44:08 We are looking at it.
44:09 I’ve actually just been to Europe iPhone back in July and attended some of the very
44:15 interesting presentations on where 3.30 is heading.
44:19 There is a lot of work ahead of us to adopt it, because there are many places which just implicitly rely on the GIL for synchronization.
44:27 We will need to identify those.
44:29 There are some interesting parts where I think we could benefit from it when it comes to our resource loading system, for example.
44:38 But yeah, it’s on our radar.
44:40 We are interested.
44:41 It’s not necessarily going to be the biggest performance improvement that we could possibly get, I think.
44:49 But once it’s there, we also have more possibility or more options to explore as well
44:54 and what we might be able to do different.
44:56 Yeah.
44:56 I imagine that AsyncIO, given how much network, Redis, etc., stuff you do, will give you a pretty mega boost
45:03 as well as just the performance of CPython has gotten so much better, right?
45:08 Yeah, we’ve gotten, like, what was it, 15% to 20% improvement across the board for most things.
45:14 Like, there’s the one-
45:16 like, string handling obviously changed.
45:18 You can’t really compare that with Python 2, because now it’s all Unicode.
45:22 That always bears a little bit of overhead compared to operating on bytes directly.
45:27 But so far, we have been very happy with the performance improvements that have gotten– that we basically
45:33 gotten for free, right?
45:35 On the topic of async I/O, though, unfortunately, we can’t really benefit from that because you
45:40 need to annotate your functions with async and so on.
45:45 And with the 2.4 million lines of Python code written without that in mind, it had become quite quickly
45:51 quite clear that, unfortunately, that one is out of the window for us.
45:56 Interesting.
45:58 We did discuss it at the time, whether it was just completely out of scope of everything that we were doing.
46:07 You know, one thing in terms of colored functions, that is one thing I just talked to Joe from the Gradient Project
46:14 about Tone.io, which is a multi-threaded async runtime in Python written in Rust, but implemented in Rust.
46:22 But it has this way of running without necessarily decorating the thread.
46:26 Anyway, maybe, I don’t know, maybe it’s interesting.
46:28 Who knows?
46:29 I attended his talk at EuroPython.
46:31 Oh, yeah, yeah.
46:32 And it looked interesting.
46:36 But on the other hand, we are doing a lot of similar things underneath the hood already.
46:40 So basically, all of the I/O already happens on multiple worker threads in the background.
46:45 Interesting.
46:46 Yeah.
46:47 Really just that we have-
46:48 I mean, the main reason why-
46:50 the main way that we facilitated Stackless back then at the new schedule and what you’ll know
46:54 is that it acts as a synchronization point to get the data from those worker threads
46:59 into the main thread that’s running Python.
47:00 And I’ve looked at Toneio a little bit, but even that is not very straightforward to integrate
47:06 in our stack, unfortunately. So doing comparisons is a bit difficult.
47:10 Yeah, honestly, and the free-threaded stuff, honestly, it scares me. I used to do tons of
47:16 concurrent programming, threads, events, signal, locks, all that kind of stuff. It’s not my code
47:22 that scares me. It’s all the libraries that are written in Python that, like you all said,
47:27 a lot of people assume the GIL was a thing and how many of those have some little latent race
47:33 condition that when they figure out what it is they’re gonna put locks on it then it’ll become
47:36 a latent lock a deadlock you know what i mean like just having faith that all the foundational
47:41 stuff has really been through the ringer of pre-threading i don’t know scares me i’m optimistic
47:46 for it and i want it to exist but it scares me yeah there’s a lot of work for sure you know also
47:50 like the other side that the single threaded use case shall not degrade in performance either
47:54 you know it’s there are many things but that being said like even with the
47:59 asynchronous programming that we that we use underneath the hood we are already
48:02 in the world of having to be careful what we put a lock on and whatnot right
48:07 for you guys you all are in this world but I think 99% of the Python people
48:13 don’t even think about locks and race conditions you know and that’s I think that’s where
48:18 we’re going to have to as a community work through the issues to make these
48:23 dependencies stable Yeah, absolutely.
48:26 There is actually– I think there is a community effort going on, as far as I remember, where-
48:32 is the URL really rb-free-threaded yet or something?
48:35 So there’s a project where they basically track all of the public libraries in PyPy
48:40 and how they behave under free-threaded Python already.
48:43 So we also– well, I keep an eye on that as well to see where that is happening and where there may be some
48:49 of our dependencies in there.
48:50 There’s the compatibility, the free-threading compatibility status checker.
48:54 It’s got the tested in CI, the PyPI release, when it was first supported, the free threading bit.
49:00 It’s very data science heavy though, isn’t it?
49:03 Yeah, I guess it’s once again the data scientists that are leading the way here.
49:06 I guess they’re also mostly benefiting from a free thread in Python, given that they are doing a lot of number crunching.
49:11 Right, exactly.
49:12 Because I think async.io, even though it may not apply to you all, does solve the problem of I got to talk to the database
49:17 a whole bunch of times concurrently or to microservices concurrently because it’s the IO bit,
49:25 but when you got to do computation, it doesn’t help you a bit.
49:28 So yeah, you’re right.
49:29 I think that’s fundamentally why.
49:30 All right, let’s talk the actual migration.
49:34 So you all started by just making sure, well, first converting everything to run
49:39 in two and three, right?
49:41 Tell us about that.
49:43 Yeah.
49:45 All right, so I’m going to take this.
49:47 So a lot of the work that kind of came our way was to take our 2.4 million lines of code that we have
49:57 and bring it forward so that we can eventually end up everything on Python 3.
50:01 What we found is, like we mentioned earlier on, we still have a lot of code that was technically 2.4, 2.5,
50:10 or using 2.4 and 2.5 kind of idioms, which are no longer suitable in Python 3.
50:15 So the very first step that we’ve taken is what we’re calling legacy to 2.7,
50:20 which is get all of those legacy bits of code up to 2.7 standard, because a lot of those changes then become more forwards compatible into 3.
50:30 We’ve actually already released and deployed the first part of that a few weeks ago.
50:36 That was around the time that we wrote that blog.
50:38 And for a lot of that, we use these Futurize scripts, all the fixes from Futurize.
50:44 And what they do is they take one particular idiom of Python and update it to Python 3.
50:51 There’s a subset of those which you can run whilst still being 2.7 compatible.
50:58 And so those are the ones that we’ve done so far.
51:00 Yeah, and you just basically got them all moved to a way that theoretically would run on Python 3
51:07 because for people who don’t know, some of the older ways of working in Python 2
51:10 were literally incompatible, like ways of declaring exceptions, handling some of the string stuff.
51:19 Yeah, the big one we found a lot of was testing membership in a dictionary.
51:24 So now you do if X in Y.
51:27 In the older Pythons, you would do if there was a has key function.
51:32 That is no longer supported in Pyth 3.
51:35 But we still have a lot of code, or we did have a lot of code that was using has key.
51:39 So we had to change all of those to using the in state in operator.
51:43 So that’s the kind of thing that we’ve kind of done in this first phase.
51:48 What we found was we were actually 94% of our code was technically Python 3 syntactically valid.
51:58 We’ve now got up to 99.8%.
52:02 So that’s syntactically valid.
52:03 It doesn’t mean it’s going to do the right thing when you put it in Python.
52:07 The classic example is division.
52:09 Being a game that does a lot of things with a lot of numbers, we have about 6,500 lines that feature a division.
52:17 some of those are going to break if we just switch to Python 3 without doing anything with them
52:23 because the way it changes from integer to float division.
52:28 So that’s going to be part of the next phase that we’re going to get into.
52:31 Yeah, that was one of the main deals.
52:33 If you have x slash y in Python 2, that would do basically integer division, truncate it.
52:40 They take the floor of it more or less, right?
52:42 And then those just become floating point numbers, which that’s terrible for like this thing equal,
52:46 equal that thing and all, it’s-
52:48 And if you’re doing a lot of damage calculations, you’ve got one ship shooting another ship,
52:54 that can make a lot of difference if suddenly your damage changes and now I should have won this fight
52:58 and now I’ve lost it because the maths is working out differently.
53:01 So that’s gonna be important for us to get right.
53:04 Yeah, absolutely.
53:05 Okay, so what next, what happened after that?
53:08 You got it working on Python 2, but syntactically Python 3?
53:12 Yeah, okay, so the next phase, which we’re now in at the moment, is we’ve forked the Python 2 branch into a Python 3 branch.
53:24 Immediately, many, many things failed.
53:26 All of our tests started showing up red and so on.
53:30 And so what we’re calling that is our Python 3 unstable.
53:33 And so from there, we need to work through, get all of our tests passing, start getting that more stable.
53:40 At the same time, we’ve got what we’re calling a 2.7 unstable branch, where we’re kind of introducing a lot of the shims and the adapters.
53:49 So this is where we can have code that’s going to maybe fork if 2 do this, if 3 do this.
53:55 The downside of that is it’s going to be a lot slower.
53:57 So that’s more of a compatibility proof versus a releasable version.
54:02 But that is going to be our middle ground for code that we can then say,
54:06 we can put this into 2.7 and make it 4.0 compatible, or this is going to be breaking compatible,
54:11 so we need to put it on the three side.
54:14 And yeah, that’s our next however many months of work.
54:18 But the other big angle of it, which is really worth calling out, is it’s not just the runtime code.
54:24 We have a lot of data that is either data at rest or data going across a network that is serialized Python.
54:33 We need to get all of that compatible.
54:34 So if we have, for example, Python 2 Alt-Star classes and we want to serialize that to send it to your Python 3 node.
54:43 Well, that’s not going to work because Python 3 is going to, what’s this?
54:46 I don’t know what to do with it.
54:47 So as well as changing runtime code, there’s a lot of data at rest and data on the network that we also need to change.
54:53 Right.
54:53 You did mention that you’re using Redis, and one of the real common things to do is just to pickle the objects
54:59 and put them into Redis.
55:01 Yeah.
55:01 But you know what?
55:02 Memory shape changes from version to version.
55:05 I’ve literally been writing a blog that’s going to come out in the next week or two, a public blog,
55:11 talking about exactly that problem that we have here.
55:14 Is that the agent blog?
55:17 Yes.
55:17 So in EVE, we have a bunch of NPCs that we call agents, nothing to do with LLM agents.
55:25 These are just characters and they have memory.
55:28 So when you talk to them, it’s going to remember that you talked to them.
55:32 And then later on, you’re going to come back and they’ll give you a mission or to go,
55:35 go and fetch me this or go and kill this.
55:38 it saves that bit of memory in the database.
55:40 The way it saves it is as a serialized Python block, so that when it comes back, we can kind of reload the memory
55:48 or reload that bit of memory from the disk.
55:51 We have those going back now for 20 years.
55:54 I think there’s about 100 gigabytes of Python memory in the DB, of agent memory in the DB in the form of pickled Python objects.
56:02 So that’s a big part we need to go through.
56:07 Because we can’t just write a database script that’s going to do those, right?
56:10 Because a SQL script isn’t going to be able to serialize and deserialize Python objects.
56:15 It has no idea, yeah.
56:16 Yeah.
56:16 Are you considering moving to something that’s version independent, like msgspec,
56:23 or Pydantic, or JSON?
56:26 Some of these things, what we want to do is change the format to probably protobuf,
56:31 because we use that for a lot of our kind of communications and storage as well,
56:37 either protobuf or flat buffer, depending on the size.
56:41 But yeah, a big part of our internal and external APIs are all built on top of protobuf.
56:47 They would probably use that.
56:48 Yeah, so that’d be really natural.
56:49 Maybe you could even just not even deserialize it, just give me the stuff and shoot it back over the network
56:54 if it wants in that format anyway, digitally, I guess.
56:57 All right, we’ll get really short on time.
56:58 Maybe just take us through the remaining arc of what you all planned out,
57:04 What’s still in the story to be told?
57:07 So, yeah, so we’ve got the data formats that we need to change.
57:12 We’ve got the runtime code.
57:14 Once we’ve got those worked out, some of the things we want to start doing are what we call a mix mode,
57:19 which is where we take our cluster of 200 nodes plus all of our clients,
57:24 all of which are 2.7 at the moment, and start to be able to selectively replace one piece of those
57:31 with the Python 3 equivalent.
57:33 So rather than doing a big deployment day where we say, okay, this is the moment, we want to incrementally do it,
57:40 kind of spread that risk and that load at the time.
57:42 Can you imagine the stress of just Oh yeah.
57:45 Pushing the whole thing.
57:46 I’ve been there for those big deployments, very stressful.
57:51 So by doing this, we can spread that risk and that load over a lot of time,
57:55 but it also means we’re developing a lot of tools to support this project that aren’t necessarily
58:01 Absolutely required, but they’re going to give us a lot better abilities.
58:06 So like this ability to deploy a kind of a heterogeneous mix of types of builds and even mixed.
58:15 Do we want to do some nodes on Linux and some on Windows, for example?
58:19 By doing these tools, that’s going to give us a lot of these options.
58:21 Yeah, give you more deployment options and so on, right?
58:24 That’s pretty interesting.
58:25 It’s an interesting consequence.
58:26 I didn’t really consider coming along, but yeah, it makes sense.
58:29 All right.
58:29 Kristinn, what else do you want to say about this?
58:31 migration from maybe from the director high level yeah it’s i mean i think it helps a lot that we
58:38 did this for frontier first because one of the problems we had when we started this is that like
58:42 we could start migrating the python code but none of the python code would really work without the
58:47 C++ modules being um importable to python as well and we could start in the python C++ modules but
58:55 like they wouldn’t really make sense without the python glue kind of tying it together so
59:00 it was really convenient to do this on a project like Frontier where the
59:05 risk of breaking things is a lot less and like the first release we did we had hourly
59:13 downtime because after like one and a half, two hours the process would just
59:18 some of the process, some process somewhere would just stop responding at all and we were
59:22 like a bit in the dark there it was like a very stressful time and it turned out to be something really really stupid so as it obviously would have but so i think
59:36 i think the team city or the build system helped us a lot obviously being a game with you can
59:42 imagine we don’t have a robust robust suite of unit tests for all of our code base like
59:49 i mean even automated tests wasn’t really like a big when we started doing this and a lot of the
59:54 code was written in the first place.
59:56 So doing like an incremental thing was kind of hard.
01:00:00 But what we did do when we started, we started just running down the build system
01:00:05 and just getting further and further into like an actual build.
01:00:09 So, yeah.
01:00:10 Yeah, it’s hard to test graphics and physics and stuff through automated tests to some degree, right?
01:00:17 And when we started, when I, at least when I started, I started talking to Thomas about this
01:00:22 when I was a technical director in Yves and then we were thinking about it for Yves
01:00:26 and it just, the entire concept of doing it just felt so, just so daunting and like big
01:00:33 that I like couldn’t imagine how we would do it.
01:00:36 We did have a good plan, I think, and that plan kind of, we more or less did that for Frontier,
01:00:44 but obviously like the stakes are higher for Yves Online.
01:00:47 So like it’s a difficult thing.
01:00:49 And I can do a big difference So a critical thing for EVE is it’s an online game,
01:00:54 so the servers need to be running every single day.
01:00:58 We can’t afford to be taking the server down for a few hours every day.
01:01:05 We need to kind of keep the product up.
01:01:06 So that’s where Frontier kind of running ahead of us was really, really helpful
01:01:09 because they could prove a lot of this out before they got to the point where they needed to be online 24-7.
01:01:15 Yeah, it gives you a lot of flexibility to do all that testing, doesn’t it?
01:01:20 Yeah, absolutely.
01:01:21 especially because the transition to Python 3 is like an all or nothing step. You can’t just put
01:01:25 like one of your– you can’t just put the base interpreter on Python 3 and then keep all your
01:01:30 native extensions to Python 2, right? You need to go all the way. Otherwise, it just doesn’t work.
01:01:36 That was actually the thing that gave– I think that was the initial very biggest concern that we
01:01:40 had, like, can we actually pull this off by doing this push out all at once? And I think it was also
01:01:45 the main reason why we did this intermediate step of going to Python 3.8, stackless Python 3.8 first,
01:01:50 because then we knew, okay, we are only changing the Python major version.
01:01:55 We’re not also changing the async library underneath the hood, which a lot of the code still has some timing dependency somewhere, right?
01:02:02 And all these things.
01:02:04 Yeah, 100%.
01:02:05 Well, congratulations for Frontier and good luck on EVE Online.
01:02:10 It sounds, honestly, it sounds like a really exciting project.
01:02:13 Just a lot of fun to work on, even if it means you can’t sleep quite as well at night for a little while.
01:02:18 It’s a little stressful.
01:02:19 It is exciting.
01:02:23 I am proud of the team for delivering it to Frontieras.
01:02:27 I’m sure I’ll be proud of Eve when they deliver this.
01:02:30 What I was fascinated by was that when I started looking into it, I wanted kind of like the war
01:02:36 stories and I didn’t really find a lot of people that had done it.
01:02:40 And I’m sure a lot of people have done a very painful Python 3 migration.
01:02:44 So my running theory was that nobody really wanted to talk about it after they had done.
01:02:49 some traumatic experience and just let it go.
01:02:54 Exactly, exactly.
01:02:55 All right, well, let’s close out the show.
01:02:58 People who are maybe trying to accomplish the same thing, they have some older code base,
01:03:02 they’re trying to migrate even from maybe like old three to new three, something like that.
01:03:07 Give them really quickly just some parting advice and also people interested in EVE online.
01:03:13 Kristi, you wanna go first?
01:03:15 So parting advice for people that wanna do this.
01:03:18 One of the things that I didn’t mention is that we actually used a contractor for a lot
01:03:23 of the stuff called Recon Digital.
01:03:26 They deserve a shout as well.
01:03:27 They did a really good job kind of migrating most of the Python code.
01:03:30 And one of the reasons for it was also that developing a live game, we kind of knew that
01:03:36 Frontier, the development team, would get immediately derailed when they had to do some
01:03:42 feature work and not supporting Python 3.
01:03:44 So that was one of the things.
01:03:46 So what I would probably argue there is like secure the resources, like make them untouchable
01:03:52 while you’re doing this migration.
01:03:53 Because once you start, you have to finish it.
01:03:55 If you pause it, it’s almost wasted effort.
01:03:58 Yeah, that’s a good point.
01:04:00 Yeah, I was going to actually say something similar to that.
01:04:03 Because we’ve got one group working on pushing ahead with the migration, we’ve still got a
01:04:08 whole bunch of developers adding features to the game every day.
01:04:11 So they’re adding those features against our 2.7.
01:04:14 And so we’re putting a lot of work into trying to make sure that they have the new code they’re
01:04:20 adding is as forward compatible as possible.
01:04:24 So a big thing we’ve rolled out this week is a new linter.
01:04:28 So we’re getting and we’re going to progressively ratchet up the strictness of it.
01:04:35 So those kind of tools to kind of do code quality inspection, I think, are going to be
01:04:39 really important for us going forward.
01:04:40 Yeah, 100% agree.
01:04:43 Yeah, very much that.
01:04:44 I would add the age-old advice of have really good test coverage.
01:04:48 It helps a lot.
01:04:50 But also, touching upon what Jamie just said, if you have a product that’s live and you need to migrate it,
01:04:56 make sure that you do this in a separate branch but have constantly ongoing migrations from the Python 2
01:05:02 side to the Python 3 side so that you discover all of these new pain points that may pop up as soon as possible.
01:05:08 This helped us on Frontier as well, where sometimes things would come in that were clearly not
01:05:14 Python 3 compatible, and then we could fix it up right away.
01:05:17 And the more frequent you do these integrations, like the less painful they are.
01:05:21 If you do it once a month, oh my god.
01:05:23 Like there was a period where we only did it like every other week, and we would basically
01:05:27 spend a whole day just resolving all of the issues.
01:05:29 So do it frequent, do it often, keep it small.
01:05:32 Keep it small.
01:05:33 Good advice.
01:05:34 Do it in small, small bite-sized bits.
01:05:36 Yeah, absolutely.
01:05:36 All right, you guys, thank you for being here.
01:05:38 Everyone listening, check out Eve Online, eveonline.com.
01:05:41 Very cool game, very cool ecosystem.
01:05:45 Check out the video I linked that says This Is Eve.
01:05:47 It’s got some great graphics to inspire you as well.
01:05:50 Bye, everyone.
01:05:50 Thank you.
01:05:51 Thank you.
01:05:52 Thank you.
01:05:53 This has been another episode of Talk Python To Me.
01:05:55 Thank you to our sponsors.
01:05:56 Be sure to check out what they’re offering.
01:05:58 It really helps support the show.
01:06:00 This episode is sponsored by Sentry’s Seer.
01:06:03 If you’re tired of debugging in the dark, give Seer a try.
01:06:06 There are plenty of AI tools that help you write code, but Sentry’s Seer is built to help you fix it when it breaks.
01:06:12 Visit talkpython.fm/century and use the code talkpython26, all one word, no spaces,
01:06:18 for $100 in Century credits.
01:06:20 And it’s also brought to you by Talk Python Courses.
01:06:24 Course completion certificates are now live.
01:06:26 If you finished a course, there’s a certificate waiting for you on your account page right now.
01:06:31 Download it as a PDF or add it to your LinkedIn profile with one click under licenses and
01:06:36 certifications.
01:06:37 Same section as your formal degrees.
01:06:40 Visit training.Talk Python.fm slash account to see what you’ve already earned.
01:06:44 If you or your team needs to learn Python, we have over 270 hours of beginner and advanced
01:06:50 courses on topics ranging from complete beginners to async code, Flask, Django, HTMX, and even
01:06:56 LLMs.
01:06:57 Best of all, there’s no subscription in sight.
01:06:59 Browse the catalog at talkpython.fm.
01:07:02 And if you’re not already subscribed to the show on your favorite podcast player, what
01:07:06 are you waiting for?
01:07:07 Just search for Python in your podcast player.
01:07:09 we should be right at the top.
01:07:10 If you enjoyed that geeky rap song, you can download the full track.
01:07:13 The link is actually in your podcast blur show notes.
01:07:16 This is your host, Michael Kennedy.
01:07:18 Thank you so much for listening.
01:07:19 I really appreciate it.
01:07:20 I’ll see you next time.
01:07:31 I thought of me.
01:07:34 Can we break the roll?
01:07:37 Upgrade the code.
01:07:39 No fear of getting old.
01:07:42 We tapped into that modern vibe overcame each storm.
01:07:46 Talk Python and me, async is the norm.
About the Author
