Wednesday, February 20, 2019

Vertically scaling product development

The title sounds like a "buzz thing", but it’s not. Let's explain this with a story. I might not be 100% accurate, I only witnessed the evolution since 2014.

From Startup to unicorn at the R&D

On a startup model, like OutSystems back in the early days, product development was done by a small number of engineers, incrementing a single product line. This is normal on a startup, the product is typically small (MVP - Minimum Viable Product), money isn’t something that a startup can spend "foolishly" and the focus is usually on "staying alive".

Engineering


As the product and the company grew, the number of engineers also increased and, to scale, we had to break it into teams, with a clear separation of responsibilities but still one product line:

Engineering


Because the product was "a kind of monolith", our releases were still not that dynamic. As time passed, the product grew to a complete offer for the customer’s digital journey. This meant that we also had to transform our teams to be responsible for a given area of the product (much like Spotify's model):

Engineering


Of course, each of these areas grew fast and we ended up having multiple teams working for the same product line:

Engineering


It started to get crowded. We needed a new strategy.

Vertical - Enablement teams
Up until now, what we saw was a kind of "horizontal growth": you add people and teams but "all at the same level". All the teams have the same goal (apart from infrastructure teams, release, etc): deliver value to external customers. This means that a normal story mapping session ends up with user stories looking a lot like these:
"As an OutSystems developer, I can..."
"As an IT Manager, I can..."
"As a Business Unit Manager, I can..."
...

The value is "driven" by external personas.

Mindshift - a new customer

To accelerate, we needed to be creative and change the way we work. You can imagine how messy a scrum of 20 people can be or 30 teams delivering for the same product line. We couldn’t go for this path and so we went for the "vertical" growth.

First, the basics
Remember back in the old days when you had to write assembly code? It worked, but we had to evolve. Then it was C, then C++ adding things like polymorphism, then virtual execution environments like Java & .NET, then... I jumped a few steps but, in essence, what we created were abstraction layers. At the end of the day, the code still gets translated to assembly and machine code, but that’s abstracted from the developers.

Abstraction Layers
Why not apply what we’ve known for decades in this scenario too? Well, it’s what we’ve done: we created an abstraction layer of... customers! And this is key: customers. The abstraction layer is not on teams, engineers, product or tools, but customers. I know this is weird, but it’s how I feel everyone should see it.

Engineering


As an enablement team, we should also be focused on delivering value to our customers. The difference here is that these customers are actually internal product teams (that ultimately add value to external customers)

Engineering


The resulting user stories of a typical story mapping session of an enablement team look a lot like this:
"As an R&D developer, I can..."
"As an R&D manager, I’m able to..."
"As an R&D experience owner, I’m able to..."
...

Looks good, but...

Heads up
There are lots of "tiny things" you must be aware when you are "enabling":
Maintenance: If an enablement team is developing code to accelerate other team’s projects and those teams are the ones delivering new features to customers, how does maintenance work? When a customer reports an issue either through support or direct feedback, how does it escalate? Enablement teams never do maintenance? Do it all? You can go for whatever strategy you want, just make sure it’s aligned with the entire organization so you don’t lose quality nor teams are attacking less critical issues.
Public releases: Who will manage them? Are the enablement teams involved in the release process? Why? Why not? Again, do whatever fits you better.
Dependencies: We don’t do project management "by the PMI standards/guidelines". We found that we have other ways of managing projects that work better in our context. But some things from PMI are very useful, even if it’s just the concept. Case in point: Gantt charts. In waterfall, they give you a great visibility on dependencies. And we need the same here. Whenever you start an enablement project, you are most likely in the critical path. So, make sure you properly align your plan with other teams to make sure your value deliveries will be on time of what other teams (remember, customers!) need.

Engineering


VMs are Value Milestones. They represent the minimum value increment possible on a given project (sort of).
Responsibilities: You don’t do enablement "just because". You are most likely working with/for other teams (customers!) and there are a lot of people interested in your project. So, it’s essential that you build a Responsibilities Assignment Matrix. Something like:

Engineering


This will help you be clear about who does what on which steps. And it doesn’t need to be about individuals or tasks, it can be Value milestones and teams.
Working methodology: Will you do scrum meetings with all the teams? Weekly syncs? Sprint reviews together or kickoff a separate project? Will people move from enablement teams and sit next to other teams? Or will this break the team? Make sure you align people that will work together so there are no loose ends or context loss.
OKR’s: They are great in these scenarios where several teams are working together because it’s easy to have shared OKR’s and keep everyone on the same page. Make sure you have them.

These are the things I can remember as concerns, but there are probably more that I can’t remember at the time of this writing.

Summing up
Enablement teams allowed us to have a kind of vertical growth around product development. These teams will probably not have "big news" to share externally, but it’s ok. The question you need to answer whenever you start an enablement project is
"Are we really delivering the value that our "new" customer needs?"

Tuesday, January 1, 2019

Service Studio Beta 101 (FAQ)

Service Studio Beta

What is it?
In essence, a stable version of Service Studio but with some additional features that are still to be released.

Wait. Stable?
Yes, a stable version. Unlike a nightly build (like Firefox Nightly), the Beta version contains the same code as the general availability version but with added features. You can expect the exact same quality in the Beta version as in the normal releases.

What's in it for me?
Two main things:
- You have access to "goodies" faster;
- You can help shape the product: your feedback will be used to improve the features before they get to general availability.

What's in this first version?
The first Beta version comes with a Forge Integration in the application list that allows you to quickly update your Forge components. From now on, there are no excuses for keeping outdated components in your factory. Also, automatic updates. Yep, this version of Service Studio Beta will be the last one you'll ever have to manually download and install! You don't have to do anything: you'll get new features automatically! You can read the quarter highlights at the OutSystems Engineering blog.

Service Studio Beta - Forge Integration


How often will I get these updates?
Whenever there are new features, there will be a new version and you'll get it. It can be daily. It can be 3 times a day. It can be once a week. Apart from the Beta version (because we want to deliver as fast as possible), you have a release plan at the OutSystems website. You can also check for the What's New page.

Will these automatic updates impact my work?
No. Service Studio automatically updates in the background and won't force you to close while you're working.

I'm in! Where can I get it?
Head over to https://www.outsystems.com/home/beta. It's the last version you'll need to manually install. Automatic updates for the win!

How do I send feedback?
Just like in the general availability version. Your feedback is precious.

Anything else I should know?
Head to this forum discussion and join us! Oh, and keep that feedback coming. Let's build a better product together!

Monday, December 31, 2018

Breaking our own monolith

If you are interested in OutSystems, you probably came across some articles from Francisco Menezes and talks from the CEO about breaking monoliths.

"A monolith is the result of building every business value and functionality into a single core application. When first building a product, a monolithic strategy has significant upsides: development moves quickly, the tech stack remains consistent, data remains in a centralized location, access to functionality is at the fingertips of every developer, and others. But as a product ages and the teams behind it get bigger, the downsides kick in. Release cycles slow considerably as changes of any kind (big or small) require a full-product release."

Well, we also had a monolith problem with the platform itself and we had to follow the same path: break it! We needed faster release cycles!

OutSystems Overview


Independent release cycles

Online services
The Mobile Apps Build Service (MABS) is a service that, in essence, abstracts a developer from all the complexity around the generation of a mobile app (XCode, SDKs, etc). This was designed, from scratch, to be loosely coupled from the rest of the platform. By following this path we got ourselves independent release cycles for MABS. MABS moves at its own speed as needed and customers get automatic updates whenever they generate mobile apps. OutSystems Insights is another piece of software at OutSystems designed from scratch with all this in mind.

Service Studio
This is a bit more tricky. Both Service Studio and the Platform Server share lots of definitions. To illustrate (and keeping it simple), imagine that we introduced a new element in the language that only Service Studio knew about. When you published it, what would happen on the platform side? So, the full decoupling is not that easy. However, there's no reason why Service Studio cannot get lots of "goodies" without requiring a platform release. So, Service Studio is scheduled to be released every week. Even further, recently, we released Service Studio Beta with automatic updates. That means you can get features in your hands automatically in no time. We release --> you get an automatic update --> new feature!

OutSystems Overview



OutSystems UI, OutSystems Now Service, Supported plugins
Well, Forge components, basically. These have a completely independent "life". The release cycle for these components is as simple as publishing them in Forge.

Lifetime
Why should Lifetime be dependent on platform releases? If there's an improvement in the mechanism to stage apps, what does that have to do with Service Studio or Platform Server? It doesn't. And Lifetime currently is on it's own release path.

These 4 above are great examples of independent release cycles at OutSystems. And this means that teams can work independently, at their own speed, releasing value faster to the hands of customers and developers.

Thursday, November 29, 2018

What will you remember?

I'll start this post with a personal story.
I was part of the team that built the Mobile Apps Build Service (MABS). This is that piece of software that allows you to generate mobile apps using OutSystems, abstracting you from all the complexity of native mobile development. This service is supposed to run 24/7, we're targeting a 99.8% Service Level Agreement (SLA).
It was 1am on a Friday. The night was going really well, just had dinner with a beautiful curly-haired, “green eyed” woman who was now naked in my bed. I was also naked, “ready to roll” (whatever you want to call it). Suddenly I got a call. The number was from OutSystems. Well, at that time of the night it could only be support teams. My number was the first one to call in case of an emergency with the service. I picked-up the phone, Android builders were stuck and customers were not able to generate mobile apps.

There I was. On one hand “fireworks”, on the other issues with Android builders.

Angel vs Devil


Can you guess what I did? What would you do?
I know, you’re thinking "f*cking jump into the bed". I didn’t. I put on some boxer shorts, went to the living room and opened AWS console to start troubleshooting. 20 minutes later, great success: service was restored and… the girl was sleeping.
This was real. It happened. And now I have a story that I’ll remember for the rest of my life. Of course I had to apologize to her and explain that decision. I would feel offended too if someone did the same to me. But I did the right thing. I had people depending on me. I’m not in it just for the money, I’m in it to be good at it. And that’s the only way to do it: Go big or go home.

When you're old, you won't remember the "normal days at work". You won't remember the 9 to 6pm normal work. You won't remember the routines. You'll remember the crazy stories, the nights when "shit hits the fan" (best team building events happen automatically when things go wrong), the applause when you do a kick-ass work. You won't remember the days you just did your job, you'll remember the days you excelled. That's where I want to be, where the adrenaline is.


"But... are you saying that we should always be in trouble or with high adrenaline"? Well, no. You would get really really really tired after some time. My worst job was the first one. I was working from 8am to 11pm, doing everything: project management (sort of ), writing code, talking with customers, writing specs... a recent graduate! Of course it didn't last long. I just held on for a year because I was too proud to quit. I learned a lot and I have lots of great stories to tell, but that was no way to live.

Nevertheless, point in case: give it all while you can. It's like sports: if a game has 90minutes, give it all you got until you can no longer stand up. Even if you can't breathe after 30min, doesn't matter. Be the best on those 30 min.

Side notes
- although my number was top1 on emergency call, the first task was always to actually gather the team. That story was the only example where I didn't and went on my own. Usually, we would all be ready to work after 5 to 10min. Amazing team spirit "when things are not working".
- I ended up getting married to the green eyed girl :)

Wednesday, May 9, 2018

Stopping chaos while still being helpful

The Small Book Of Few Big Rules


Be helpful.
This is one of the rules you'll find in the small book of few big rules. This small book contains the core values to be successful at OutSystems. I guess it's actually a key part why the CEO says that "80% of people will become great when in the right environment". And people take these small rules *very* seriously at OutSystems.You can go around the office and feel this, like if it was something you could touch. People guide themselves by this book, it's amazing. Now, focusing on the help rule:

People at OutSystems are open to helping you. Use this to your advantage as much as possible. And do the same yourself. Offer help even if it is not in your job description. We believe you should extend yourself outside the boundaries of your work and into other functions to get the job done. Remain aligned with the Ultimate Goals when offering assistance. When it is your turn to look for help you will nd that people will quickly reciprocate.

Pretty simple right? In essence, it is. However, my team actually struggled with this rule. We helped too much, in a chaotic way. How is this possible? The Mobile Team is highly exposed to pretty much everyone, from support to customers, to peers, to...It's the main entry point in the R&D for anything you can think about related with mobile. Remember that OutSystems 10 brought Low-Code development to mobile apps. But, of course, it wasn't all done by the mobile team. We had literally hundreds of people involved from dozens of teams (IDE, FrontEnd Teams, Core Teams, you name it). Think of us as a group of pretty organized ants, carrying blocks of "value" instead of sugar to a big platform.

OutSystems 10


At some point, a combination of this exposure with our strong feeling about the small rules became toxic. Too many requests together with our willingness to help resulted in a huge growth pain. We created a monster. That's not necessarily good in a R&D department and can highly impact value delivery to customers.


Too many channels

OutSystems Mobile Team


First there was skype. Then Hangouts. Then... Without realizing, we had incoming help requests coming from all sources of communication channels. People would jump to our spot in the office, interrupting all the time in search for help. Some would "skype" a team member privately. Some with hangouts. Feedback would come through mail, support cases through JIRA, mail... Slack, which appeared later was the final "stone". A public #rd-mobile-team slack channel you could use to get help? Sounds awesome, except you would divert the attention of an entire team. And private messages, now on slack too?

Monkey User - Focus

Click to enlarge - source: http://www.monkeyuser.com/2018/focus


It started to be too easy to reach the team. All team (and more) distracted because someone entered the team channel to ask "which plugins in forge are supported?" Or... Of course, we replied to everyone on every channel because we wanted to be helpful. But it was taking a huge hit on the team performance and focus.


Act!

What to do? Well, what we do best at OutSystems: change!

Focus

Click to enlarge


Back to the drawing board. First step? Notify management that we were not working properly and ask for their input. Ideas! "Call me a whiner any day". And they did gave us a great starting point. We merged their ideas with our own and we had a plan.


Executing the plan

"Siblings"

We're a fast growing company that's in constant mutation. Everyone is open minded and always willing to change. And top management supports change and encourages autonomy of people to apply these changes. So, let's do what best fits our team. But first... how do other teams around us work? How have they dealt with this issue in the past? We talked with the most similar to our team and got great insights. Again, we would to whatever would best fit our working methodology, but ideas are welcome.

Measure

"You can't manage what you don't understand". We started with a small impact analysis: how much was it costing to be so helpful? Impact on team speed? Also, a small risk analysis: what are the risks of helping through private channels? What if some erroneous information ends up in a support case and a customer gets the wrong expectations? Well... we were surprised with so many risks that we didn't think of. We never stopped for a moment to look into this issue. Hell, we're growing at the speed of light!

Action items

Write down action items and owners. By action items we're saying what must be done to improve and the owner is someone that will assure things will happen (not necessarily the person who implements it). This will get things rolling.

The "bad boys"

It's so easy to just open a chat window and ask things that it's the most dangerous part of this equation. So, our first targets were the chatting channels:

OutSystems Mobile Team


  - Reduce scope: No answers through Skype or Hangouts. Our official, internal, company wide tool is Slack. Period.
  - Redirect from private to public: private Slack chats shouldn't be used to ask for help unless you specifically want to talk to that person. So, ask in the team channel. Not only you'll have proper help but everyone in the channel will have the knowledge (E.g.: support engineers, product owners, etc). If someone has the same question again in the future, it will be easy to answer and everyone can do it. So...

OutSystems Mobile Team


Still a bad boy

Ok, so, it's set: internally, anyone should just go to the #rd-mobile-team slack channel for help (help is different from support). But we still don't want everyone in the team to be distracted.

OutSystems Mobile Team


  - Don't use @channel and @here unless it's critical. This alerts everyone in the channel that has notifications active for that channel. Someone will answer when it makes sense.
  - Maintenance only! Only the person in the team doing maintenance can reply unless it's critical. Block interruptions from the remaining team members and make sure 2 individuals aren't answering to the same request.
  - Wait: Yep, probably no one from the team will reply in the first 15-20 minutes. That's due to two main reasons:
    1) Most people ask without giving it a good previous thought (because it's so easy to just ask and some are just "rubber ducking")
    2) Maybe someone from support or similar will be in the channel with all the context and will help even better.

OutSystems Mobile Team


The not so bad boys

OutSystems Mobile Team


Slack is great to ask for help. But sometimes people just want to report a bug or provide feedback. Slack is not good for that. Mail the same. Why? We'll have to grab that feedback and create issues on our backlog. So, we opted-in for the internal tool that was already in use by other teams for this purpose. It's called Sauron and it's, in essence, a centralized way for all teams to receive feedback, prioritize and manage that feedback. Yep, we look at everything you provide as feedback (E.g.: Service Studio). Again, only the person doing maintenance can look at this tool for a given sprint. After the initial analysis, the maintenance member can decide all sorts of things, like the priority, impact, risk, etc. It can also send to another team if it fits or immediately to team backlog (if it's something that the team should work right away).

OutSystems Mobile Team


Customer success is our success

Ideally, we wouldn't have any support cases. And we wouldn't have support at all. The product would be so good and so intuitive for everyone that no customer would need help. We'll get there, dream high.
However, while we don't reach the sky, we do want to make sure every customer gets all the attention he deserves. So, we treat support cases in a different way. Whenever a customer opens a ticket and it escalates to R&D, we have to make sure we react fast. So, if support needs to escalate they will automatically open an issue to the team in our "special board", we get notified and the maintenance person will quickly look into it. Support provides all the info through that issue and we iterate really fast with them. It's a high speed lane to us.

OutSystems Mobile Team


All customer related issues coming through proper support channel has a huge advantage: centralized knowledge in support. Only strange issues require R&D escalation nowadays as support gets more and more expertise on the product. And all customer communication is done in one place.

OutSystems Mobile Team


Be helpful... Be helpful... Be helpful!

It's a big rule right? And we love it! So, we still want to help everyone and live, on our spot, is still an option. Just pick a good time for it (middle of the morning is not a good time, focus days are a nooooo please - unless you bring food) and, if possible, go directly to the maintenance guy. It's the one with the fire extinguisher.

OutSystems Mobile Team


Result

Summing up, all this seems simple and pretty obvious right? Well, it is. But the "monster" didn't show up from one moment to the other. It was something that slowly appeared without us realizing. And it was a huge release by OutSystems, with everyone, including OutSystems developers still absorbing all the information. You can see that we still have a lot of communication channels. But it's ok. The issue was chaos. We have brought some sanity to the team and we can help in a much effective way. We are a happier team.

OutSystems Mobile Team


Sorry about the drawings. My skills are not that good and this was done on a plane. But hey, communicate to be understood :)

Friday, April 27, 2018

Funny code comments – zen

We've all been there: that moment you're so tired that you just do a crappy code and you place a comment admitting it's not good. That funny comment because you know someone will look to it later and laugh. Below is a compilation from several sources, grab a beer and have fun!


OutSystems

OutSystems

OutSystems

OutSystems

OutSystems

OutSystems


Shared in stackoverflow






Note: These were compiled from this Stackoverflow thread and this post.


.NET Core

We're doomed! (Source: ConcurrentExclusiveSchedulerPair.cs)


Maintenance (Source: Parallel.cs)


Zombies... (Source: Scheduling.cs)


Curve ball (Source: DebugTypeDescriptor.cs)


Legacy (Source: isymwrappercore.cs)


I can't prove it! (Source: EventLogInternal.cs)


Don't touch (Source: compareinfo.cs)


I don't understand (Source: methodbuilder.cs)


Hooker? (Source: console.cs)



Note: These are from a post I wrote in 2014
The .NET Core is OpenSource.