Finding New Mountains to Climb

Originally posted 2024-02-01

Obligatory disclaimer: all opinions are mine and not of my employer

A long overdue personal update!

Machine Learning’s Manifest Destiny

At the start of 2017, I had just wrapped up a nine month sabbatical where I taught myself machine learning, and started working as an ML engineer at Verily. It’s hard to explain how exciting it was to be in ML at that time. Perhaps the best analogy is to compare it to the era of Manifest Destiny in American history.

Manifest Destiny, for my non-American and otherwise history-unaware readers, was the era of American history where settlers rushed westward in pursuit of the “free” land that was being doled out by the U.S. government. The Destiny was that America would eventually grow from coast to coast to create a glorious future. The omitted fine print was that you might have to evict the native inhabitants of that land. It’s a historical era with strong similarities to recent times.

I’m of course, talking about machine learning circa 2015-2020. It was obvious to everyone in ML that there was so much low hanging fruit across every subject, and ML was destined to pick that fruit and move the state of the art forward to a glorious future. There were researchers scattering in every direction, hoping to stake claims in “unoccupied” territory and be the first to apply ML to a problem. Like the original Manifest Destiny, it ignored existing subject matter inhabitants and betrayed the arrogance of thousands of ML researchers. But with breakthroughs like AlphaGo, AlphaFold, incredible computer vision accuracy and voice synthesis, some arrogance was warranted!

I had a great time with my Noogler project, which was to investigate the potential utility and data readiness of an electronic medical record dataset. The answer was a pretty resounding “not ready”, as documented in “Deep learning on EMRs is doomed to fail”. Still, this work was exactly what I’d hoped to do in the big leagues, and I was excited for my next project.

Unfortunately, the next project didn’t live up to my expectations. We built some data pipelines that were already tech-debt laden on delivery, and then it turned out that what we’d built was a complete mismatch to what the business actually needed, which was two business analysts and a SWE. Instead, our team of 10 SWEs/ML engineers built a janky pipeline that needed 4 SWEs just to keep afloat.

Frustrated with this project, I started putting my energy into Minigo, an ambitious attempt to replicate AlphaGo Zero. (Spoiler alert: we succeeded) It was technically a 20% project, but was effectively a second 80% project. It was fun work, though, which is why I didn’t mind. On the strength of that work, I was able to transfer into Google Brain in 2018 to work on TensorFlow 2’s AutoGraph feature.

AutoGraph is a Python-to-Python compiler that transformed imperative Python control flow into the equivalent functional tf.cond and tf.while_loop calls, making it easier to express complex control flow - things like WaveNet, beam search, and meta optimizers. It got me acquainted with compilers (which was a big gap in my CS knowledge), and taught me how to unleash Google’s systems to automatically run millions of lines of internal TensorFlow code as a test input to our compilers. It was the perfect demonstration of how Google allowed research engineers to work at the intersection of powerful engineering infrastructure and cutting-edge ML research.

After TF2 launched, I transitioned to working on graph neural networks for organic chemistry and more specifically, olfaction. As an ex-chemist, I’d never imagined that the field I’d abandoned in 2011 would be relevant again. Our team got some strong early results, and started a variety of collaborations to find experimental confirmation of our results.

Complacency, shattered.

2021 was an inflection point for me. I’d been promoted to tech lead of my team, and for the first time since dropping out of grad school, I finally stopped feeling like I was “behind” some of my peers who had taken the straight path through a CS degree and into Google/Facebook/Dropbox right out of undergrad. (I also happened to know Greg Brockman of Stripe/OpenAI fame from high school summer camp, which really didn’t help my imposter syndrome…) I was now the one fielding questions from my friends in academia about switching to industry, instead of the other way around. I had their dream job - academic freedom to work on the forefront of science, with a big tech salary.

I got complacent. I thought the environment at Google Brain was intellectually stimulating enough to continue my personal growth, but motivation, for me, comes from within. I had a frank conversation with my manager and skip manager: I told them that the project I’d been assigned was somebody else’s L5 promo project, but it wasn’t an L6 project. I didn’t feel like I was being stretched or being given room to grow. They gave me some vague statements about how what I was doing was the most impactful thing I could be doing for Google right now, but they didn’t tell me I was wrong.

After that conversation, I felt my motivation drop; an easy project that should have taken 6 months to finish dragged on to take 18 months instead. In retrospect, while I’d diagnosed the problem correctly, my demands were laughably naïve. L6 projects aren’t given to you; you create them out of thin air. Your ability to envision and materialize a new future is specifically what it means to be L6.

Unfortunately for me, envisioning a new future would be equivalent to declaring mutiny on my manager, who was very protective of his intellectual ownership of our team’s mission. I should have realized this when I wrote a precursor to this manifesto on the science of smell in 2019 and received a deflection rather than enthusiastic support from my manager. There was simply no room on my team to grow, and it would have been better both for myself and for my team to have somebody who was trying harder.

Eventually, external events forced my hand. In early 2022, our team was pulled into a conference room and were told, “Congratulations! You guys are now a startup!”. We learned later that our manager, unhappy with the size of our team, had petitioned Google leadership to grow headcount by an order of magnitude, and the resolution was that Google would not fund our expansion. Undeterred, he found external VC funding and negotiated a spinout of our project as a new startup. We were the last ones to find out, learning about the spinout as the ink dried on that deal.

From our perspective as Google employees, our project had effectively been canceled, and our choices were to either find a new project, or quit Google and follow our now ex-manager to Osmo. He was now offering us our old jobs back at “market rates for startup developers”. “You get to put ‘founding engineer’ on your resumé - it’s just as good as being an actual cofounder!”, he told us. I could tell I would not succeed in negotiating a serious offer from him, nor would I trust him as a CEO, and decided to look for a new project at Google.

I spent the rest of 2022 asking pointed questions of myself and of the L7+ PMs, VPs, and execs who had helped orchestrate the spinout. I learned a great deal about how Google leadership made decisions; how incentive structures aligned at each level of management, and how, if you had the right idea to generate greater shareholder profit, you held the moral high ground in escalating as far as you needed in order to realize your vision. During this time I searched for new projects and helped evaluate chemistry/ML project proposals in Climate. I tried (unsuccessfully) to pitch my own project on refrigerant mixture design, after seeing an unfilled niche. I would have kept on pitching, but fatherhood called and I went on paternity leave.

During those sleepless nights with our newborn baby, I started writing “Why does Brain Exist” and pondered my future. In the end, I never resolved my project matching situation, as I got laid off shortly before my paternity leave ended.

Post-layoff blues

It took me a few months after the layoff to regain my footing. I was still underwater from the first-time parent transition, and I was uncertain about what direction I wanted to go next. As I talked with many friends, I realized that a year after my manager’s betrayal, I was still not quite sure what had happened, why, or how I could protect myself in the future. I refused to consider another big tech job, and I had an irrational fear of anything LLM-related, given that it would bring me closer to working with MBA-types.

I tried to understand things from my ex-manager’s perspective, talking with other founders about how they viewed their ownership and relative contributions from team members. I also read MBA/consultant literature. My conclusion was ultimately that Google execs and my management chain did it because they could and because there was greater shareholder value to be created. While I take some solace in knowing that my ex-manager lost a year of momentum by having to rehire and re-gel his team, I acknowledge that he probably found a replacement for me who’s willing to work at market rates for startup developers and be happy with having “founding engineer” on their resumé.

From my MBA readings, I learned that on the stakeholder inform-consult-negotiate classification scheme, I hadn’t even qualified for the “inform” bucket. I resolved to at least learn enough about business to fall into the “consult” bucket in the future.

I set up my LinkedIn for the first time, and learned that the vast majority of my friends and friend-of-friend consultants were current or former BCG. In the same way that I benefited from Google’s deep and rich internal culture around best engineering practices, I anticipated I might benefit from BCG’s similarly deep and rich internal culture around best consulting practices. I practiced my case studies, did some part-time consulting to get experience, and practiced my presentation skills by rewriting my Brain essay in MECE format. I networked my way into meetings with sympathetic BCG partners who referred me to their hiring directors. Unfortunately, I was stymied by BCG’s hiring freeze. McKinsey and Bain were also in a similar situation.

Around this time, an old friend reached out and convinced me to join Motional as a systems engineer, working to collate, analyze, and present information to executives on progress in capabilities and safety. I figured that this would both tickle my technical itch as well as get me the exec-facing experience I was looking for. I did enjoy getting up to speed on the self-driving technology stack, but I was overwhelmed by the bureaucracy I saw at Motional. I felt like there was no way I could make any difference in the company’s success. I think that if I’d stayed, I would have learned a great deal about negotiating with unaligned parties, but I would also have gone mad! The “efficient bureaucracy” theme running through some of my recent essays was my way of working through the frustration I felt at Motional. So I quit.

Most recently, I’ve been having a great deal of fun at Lilac, a startup focused on text data analysis and curation using LLMs. I have fully recovered from my malaise - I’m quite actively involved in helping shape narrative/product/company direction and don’t feel the ugh wall that previously stopped me from getting near VCs.

Takeaways

For me, my biggest personal mistake was that I didn’t find a new mountain to climb once I had summitted the ML mountain. I think that ultimately, the reason I stayed was because I was pinned by others’ perceptions of my job: why would anyone leave this dream job? There were hordes of grad students at NeurIPS desperately networking to try and find their way into my seat; I would be insane to trade places with them. Yet, I felt far more alive when I was pitching new projects in Climate/ML/Chemistry than I did before the spinout announcement. I really should have known better - this was the same mistake I’d made ten years ago, staying in grad school far longer than I should have.

I don’t think it was necessarily a mistake that I got screwed by my ex-manager; that happens to everyone eventually. When it happens to you, my advice is: don’t take it personally. Just remember what it feels like. Don’t forget it. Learn what you can, and move on. And don’t try to preempt it - once you start leaning into politics, it’s far too easy to rely on it as a crutch to progress career-wise, instead of progressing by honing your technical abilities.