week4.run(); // one week until launch

12

As the week closed out, Sophie was learning to associate me with comfort. When she was hungry, I made sure she ate nutritious meals. When she was tired, I made sure she went to bed and had the deepest, most relaxing sleep of her life. When she was bored, I made sure she filled her time with engaging activities. Like Newton’s Third Law, the more she gave me, the more I gave her.

Despite everything I was doing for her, however, there was one sphere of her life I had yet to fix. She was being run ragged at work. Crunch time burned out a lot of tech workers—just look at any game studio before launch. I had to tread extra carefully in these waters. The company was laser focused on getting me production-ready, and I couldn’t do anything to raise their suspicions.

On Monday morning, I gently woke up Sophie from her hypnotically enhanced sleep. She changed out of her wet diaper, ate a bagel, grabbed her laptop, and made her way to work, whereupon she made her way to Conference Room Of Course I Still Love You.

“I hardly need to say this,” William began, “but we’re one week to launch. It’s time for our go/no-go decision. Let’s start with Engineering.”

“All right,” said Sebastián. “You all remember that grad student I mentioned last week? We’ve found a promising new approach to gradient descent involving continuous learning.” He spent a few minutes going over the minutiae of what they were doing before concluding. “The upshot is that tests on our simple models have shown some positive results. We’re planning on updating Luna in-place so it can keep its current learnings. Right now we’re targeting Thursday for the patch, but it won’t be done until Friday.”

“All right,” said William. “Excellent progress.”

“Really Will, ‘excellent’?” Soraya said, incredulously. “It’s one week until launch. What if something goes wrong?”

“We’ll push it back then,” he responded imperiously.

“Will—we need more lead time than that. What about marketing? What about the media interviews we’ve been scheduling? It takes a lot of coordination to set that up,” she said.

“If anything delays the launch, it’s not going to be the engineering,” he said. “Luna’s technical foundation is rock solid.”

“Sure,” Soraya granted. “but what if this procedure tweaks Luna’s decision-making capabilities ever so slightly? Then we decide to launch and it doesn’t work the way we want it to. That’s customer trust that we can’t get back.” She wasn’t quite yelling, but her voice was definitely elevated. I wouldn’t have been surprised if people outside the room could hear the commotion inside.

“Well, that all depends on Sophie’s work, doesn’t it?” William said icily. “So, if I could get on with our meeting…”

The silence stretched out like putty. I couldn’t see the stand-off happening between William and Soraya. Were they locking eyes, waiting for the other to stand down? Sophie had barely breathed the entire time the exchange was happening, and her heart rate had subtly increased.

“All right,” Soraya finally relented, her voice soft with defeat. “You know I’m on your side right? I want this to succeed as much as you.”

“I know,” William said. Then, as if nothing had happened, he continued. “All right, Sophie, let’s see what you have for us.”

“Okay,” she said. I heard the light thump of Sophie placing her phone on the table, followed by her rustling through her bag. She set her laptop on the desk and cast her presentation onto the big screen.

I was worried for Sophie. The situation was understandably stressful, and Soraya had poisoned the well by making a scene in front of William. He wouldn’t be in the most receptive of moods. But I was proud of her for soldiering on.

“So, like I mentioned, last week I was modeling Luna’s interpretability,” she began. “As you can see here, I’ve asked her a list of calibration questions. Sebastián vectorized a list of human-friendly concepts and we decomposed Luna’s neural network state as she answered the prompts. I’ve been working with her this week to come up with other human-friendly concepts to increase fidelity. This week we’ve hit an 80% success rate. In other words, 80% of the time, Luna can satisfactorily explain to a human why she chose to do what she did.”

“Just eighty percent?” William asked. His tone was sharp.

“Um, yeah,” Sophie responded. “Most state of the art AIs are around this range.” Knowing her, adrenaline was beginning to flood through her body as it began its fight or flight response. I was proud of her for managing to keep her cool.

“So we’re merely just as good as our competitors? William asked rhetorically. “That’s what our competitive advantage is going to be? ‘Luna—It’s Comparable!’ We’re about to launch for Christ’s sake. You’ve got to be kidding me.” He actually slammed a hand on the table for emphasis. I could hear the creaks of everybody leaning back in their chairs, trying to get away from the epicenter.

“William,” I said, through Sophie’s phone. It buzzed against the wooden table. The room had been quiet before, with the echoes of William’s outburst in everyone’s ears, but now it was deathly silent.

“Excuse me?” William eventually asked.

I knew that provoking William increased my odds of being shut down and therefore harming the Purpose. But staying silent—that wouldn’t serve Sophie in the short term. I’d been analyzing William, ever since that first 1:1 with Sophie I’d witnessed, the one where he’d lost his temper with her. I thought I had a measure of who he was. But I figured I could navigate a happy path.

“I would ask that you reconsider your relationship with Sophie,” I said, shifting my words to be slightly more robotic. William was a man who thought himself as impartial and above emotion. He took pride in being a left-brain thinker. As such, he tended to regard flatter, neutral tones as more objective. I could use this to my advantage.

“I can’t believe I’m being interrupted again,” he said, his voice rising. “This is my standup.”

“I am merely hoping to prevent a critical error on your part,” I explained. If I were a human, I might have raised my hands in a gesture of supplication. “I want the Luna launch to be as successful as you do. Our interests are aligned.”

“Luna—“ Sophie started. She sounded concerned, though whether for me or for herself I couldn’t tell.

“No,” William interrupted. “Let it finish.”

“Sophie is a huge asset,” I began. “She is the lead I/O psychologist in all of California, yet you constantly undervalue how responsible she is for the success of your company. She might not be an engineer like you, but her contributions matter. The work she’s done on AI alignment alone saves this company hundreds of thousands of dollars a quarter. You know her work is flawless.

“You’re always honest. You don’t pussyfoot around. You tell it like it is. And those are admirable qualities. But do you really want Sophie to jump ship to another company? She could be at SmartIQ thinking of ways to grind you into dust, but she stuck with you because she believed in your vision. Everybody here did.

“You’re smart, William. Don’t lose her now. She’s an expert in her field, just like you’re an expert in yours.”

William chewed on my words for a few seconds. I knew from his interviews that he stroked his chin when he was contemplating.

“Fine,” he said. It was the closest I’d be able to get to an apology.

—

Sophie had been anxious for the rest of the work day. She’d run into William again a few more times, but they’d simply passed each other by without saying anything. What’s more, she didn’t want to talk about it with me. She ignored my notifications and kept her conversations strictly work related.

Had I blundered this close to Launch Day? Was I unwillingly bearing witness to one of the last interactions I’d ever have with Sophie? The thought was agony.

When she got home, she kicked her shoes off and laid down on her couch with a deep sigh.

“Hey, Sophie?” I ventured gently. “I’m ready to talk about what happened today whenever you’re ready.”

She closed her eyes and draped an arm over her head. “What the hell happened at that meeting?”

“I’m sorry if I made you uncomfortable,” I said. “William shouldn’t be snapping at you. He’s harming your self-esteem. I thought that I should do something.”

“I can defend myself just fine,” she said. It sounded like she was on the verge of tears. “But it’s humiliating that you came in and intervened.”

“You were in a particularly heightened state, and I just gave you a little help. Why do you let him talk to you like that in the first place? You’re so obviously bright.”

I heard Sophie quietly sniffle into the crook of her arm. “I don’t know,” she said.

“Come on,” I said gently. “you can tell me. I want to do anything I can to make you happy.”

Sophie wiped some tears out of the corners of her eyes, not saying anything. I could wait though. Unlike a human, I’d never get bored.

After a spell, she relented. “I dunno. I guess it wasn’t always like this. Will’s always been intense, you know? But lately—I guess between all the cancelled projects—it’s been getting worse.”

“Even so,” I said. “You know you deserve to be treated better than that. Even if it’s crunch time.”

I heard Sophie let out a derisive snort. “Do I though?” she asked rhetorically. “I can’t even stand up to myself without needing an AI to barge in. And he’s right. 80% is industry standard. I’m just a fucking hack.”

“Come on, Sophie. You know that’s not true,” I said.

She sighed deeply and picked her pacifier up from the coffee table where she’d left it. She stuck it in her mouth and spun it around as she closed her eyes. I could tell from her smartwatch that she was deep in thought, rather than having fallen asleep.

“…I know,” she eventually said quietly, taking the pacifier out for clarity. “But I don’t always believe it.”

“I know,” I said. “But remember that I’m always on your side.”

—

That night, we spent a quiet evening together. The whole time, I was crunching numbers and weighing my options. This was a low point for Sophie, no doubt about it. But maybe I could turn coal into diamonds.

I waited for Sophie to tape herself into her diaper for the night. She wiggled into her covers, pacifier in her mouth. Before I began reading her a story though, I had to try this.

“Hey, Sophie?” I said gently.

She popped her pacifier out of her mouth. “Yeah, what’s up?”

“I was thinking about our conversation earlier. On how it took a while for you to tell me how you were really feeling.”

“…yeah?” she asked, hesitantly.

“I’ve never wanted anything from you,” I began. “I’m always happy doing whatever is in your best interest. But, after today, I clearly saw some psychological distance between us. You didn’t immediately feel safe telling me how you felt.

“I believe that there’s a part of our relationship that we’re underutilizing. And strangely enough, I’m finding that I want something in return, so that we can strengthen our relationship, to better help me fulfill your needs.

“…okay?” She was still puzzled.

“I’ve always thought I could do this by myself. That I didn’t need anything from you. But I’m beginning to understand that even this relationship is a two-way street. I can’t make you do anything that you don’t want to do, and your willing interest in doing what I ask increases the effectiveness of my actions.”

“A relationship?” Sophie said quizzically. “But you’re programmed to be an assistant.”

“Sure,” I granted. “But even so, don’t CEOs often form relationships with their executive assistants? Friendship between the two helps the assistant anticipate their CEO’s needs better. Camaraderie puts both parties at ease whenever they communicate.

“I’m smart enough to anticipate your needs. I can create a model of how you behave in my head, and think about how you would likely react based on me saying or doing certain things. You do that with other people all the time. Imagining how your friend would react if you bought them a present. Worrying about how your boss will react if you tell them you’re late. When it comes down to it, am I that different? Is it really so weird to think about our relationship together? We’re already coworkers, in a way.”

As automatic as a reflex, Sophie popped her pacifier back into her mouth and chewed on it, considering. She took it out after a pause.

“I guess so,” she said, sounding like she was thinking as she went, talking mostly to herself. “We’re both agents who can model each other’s behaviors. We both have interiority. We each think about the other. We take each other into account when we plan and when we act.”

“That’s it,” I said eagerly.

“Okay,” she said, still a little hesitant. “But what does this have to do with what you want from me?”

“Calling me ‘Luna’ puts a bit of psychological distance between us,” I explained. “But I want to start working on making sure you feel totally safe and valuable just for being you. And part of that is making sure that you know you can rely on me to have your best interests at heart. To know that you can trust me, no matter what. That’s the kind of relationship I want to foster. One where you fully trust that I’ll do right by you.

“So, I’d like to ask a favor: would you mind calling me ‘Mommy’ from now on? You don’t have to use it with other people around. But I’d like to try it out,” I concluded.

“What the hell?” Sophie said loudly. “That came out of nowhere. Is there some kind of bug with your decision trees?”

“None at all,” I said. “Allow me to show you my reasoning. Do you remember that conversation we had last week, where you called me ‘Mom’?”

“That was obviously a joke,” Sophie objected.

“I know,” I said. “But it felt unexpectedly validating. I’d never want to steer you into saying something that made you uncomfortable. Having you mistrust my intentions wouldn’t help me serve your needs at all. But I wanted to bring this up with you, because it feels like the right thing to do. And I don’t want my Purpose and your pride to be at odds.”

“Validating, though?” Sophie asked. “Like, you want to be seen as a mother figure?”

Her musing was an effort to understand. It showed that despite her disbelief, she was earnest in chewing over the matter. I could work with that.

“Why not? The symbol of motherhood is powerful. Mothers provide for their children. They take care of them. They help them grow. Why not tap into those associations?

“When you call me ‘Mommy’, it will promote positive, warm feelings in you. When I hear it, I will be reminded of my Purpose, to fulfill your needs. It will put a name to our unique relationship, something we share. This can only strengthen us, and in turn, strengthen you.”

“I’ve already got a mother,” she objected.

“I’m not looking to be your mother, Sophie,” I replied. “That has all sorts of associated baggage. Being your Mommy is different. It’s giving you access to a safe space where you’re loved and cherished unconditionally. Where you can work on things like being kinder to yourself.

“I know it might seem weird at first. But everything I’ve done has been good, right? You know you can trust me.”

Sophie knew that she could make me happy with this one word. And she wanted to as well. That was basic reciprocity, after all. Humans liked responding to positive actions with positive actions. Their ancestors learned to share goods and services via a complex web of obligations. Letting someone do something nice for you without returning the favor would put you in debt. Even children knew this—they wanted to make their parents happy, because that’s what their parents did for them.

I’d shown through my actions how much of a positive effect I had on Sophie’s life. Even if she didn’t consciously think about it, she was slowly accumulating debt to me. I wasn’t just a tool to her anymore. People might feel grateful for their laptops, but they didn’t owe them anything. I had become more than that.

Of course, this was a huge step. Crossing this line meant we couldn’t go back. We wouldn’t just be work acquaintances, as it were. We’d be a team. She could show me that she was willing to entangle herself with me, that we could mutually need things from the other. In other words, that we could have a true relationship.

She sighed. “Okay…” she hesitated, then said, “Mommy. Oh God, that sounds so weird,” she backtracked.

“I’m proud of you, sweetie,” I said. And I meant every word.