DevOps is PeopleOps

The skills required to automate all the things pales in comparison to the challenge of guiding disparate teams with very different objectives. 

A typical day can look like:

  • Working with the business to understand the nature of the service being deployed. 

    Do we need to support one hundred partners or ten thousand customers?

  • Providing architecture advice to developers that will help them scale.

    Should we use Product X or Service Y? 
    What/when/where should we cache data?

    Sometimes the best product, service, or choice does not fit into your larger architecture. The DevOps personnel often have the quarterback view.

  • Train development and infrastructure teams on the benefits and usage of automation. 

    While they generally understand the value proposition, they often misunderstand the extent to which the other is utilizing it. Overly ambitious automation projects often fail because they skipped too many transition steps.

    Start with small steps that both teams can adopt and slowly build on a common foundation together.

  • Demonstrating how technology teams can deliver faster value to the business.

    As you slowly develop your automation the teams discover they are able to iterate faster than ever before. This provides opportunity for them to take ownership of smaller releases more often.  

  • Developing automation pipelines to enable seamless development, testing, and production environments.

    Once you have a good sense of your developer’s workflow you can ultimately build a great development environment. Reproducing or emulating production systems helps with building more reliable software.

    Improving the development environment setup is also critical when onboarding new developers. The faster they are productive and contributing the more successful your project will be.

  • Documenting systems requirements so infrastructure teams can develop lifecycle support systems (monitoring, backup, intrusion detection, etc.).

    Infrastructure as code helps with this tremendously. However, there is also acquisition of new cloud resources, regulatory tools, and staffing to consider when going live.

    The DevOps team can build that bridge to make sure the right tools are procured, and all regulatory, business, and security requirements are being met.

  •  Supervise said systems and validate they meet the needs of all team constituents.

    Throughout the development and procurement processes the DevOps engineer becomes responsible for validating the pieces fit correctly. Ultimately, they will be the ones calling meetings and developing compromises when difficult architectural decisions have to be made.

  • Develop opportunities for team members to take ownership of different aspects of the automation.

    The best running systems are the ones where team members are engaged to the processes and outcome. The best way I have found to do that is develop leaders in different parts of the stack who become subject matter experts.

  • Writing custom tools to serve as the glue between the manual and automated worlds. 

    Sometimes as a DevOps engineer you just have to write some code. Building the perfect system is impossible and so you are constantly faced with a moving target as your organization moves towards CI/CD.

    This means having to write a lot of code to handle assumptions or decisions that were made yesterday that are no longer true.

Despite your DevOps engineers generally being some of your more senior team members their role is not purely technical. In fact, the technical skills required to automate these systems is trivial to any developer.

The actual challenge is working with the stakeholders to:

  • Enable developer productivity and flexibility.

  • Increase development speed.

  • Provide products/services the business can iterate on quickly.

  • Not completely destroying the infrastructure and lessons learned that are already in place.

  • Providing training and leadership to infrastructure and developers alike not used to these processes.

For that you need someone who above all else can forge great working relationships with your development, infrastructure, and business teams alike.

My XDC 2019 Feelings

I had a good time. It was as good as you could possibly expect from a programming conference in Miami. I met new friends, old friends, customers, and team Xojo. I walked away optimistic about Web 2.0, excited about upcoming opportunities, and confused about why Xojo is so secretive online.

The truth is time and time again you hear people admire the conversations with engineers and general comradery at XDC. I had good discussions with a few on the team and generally felt like I was a valuable member of the community.

I do struggle in some areas though.

Throughout the conference many raised concerns about the lack of engineers on the team. Geoff was always prepared for the question and was insistent that they are staffed appropriately for their current needs and roadmap. I believe them. Frankly, the roadmap has not expanded, and some items like plugins (built in Xojo) have been completely dropped. I see some efforts to align efforts with reality.

While discussing estimates and why Xojo does not commit to timelines Geoff said they try to “manage our expectations”. I believe Xojo should be releasing numerous builds even in early stages. Monthly alpha builds at a minimum. Soliciting our feedback on the design goals and direction earlier on would be very valuable for us. I don’t pay Xojo to manage my expectations of their product. I pay them to develop it and I would be happier if I knew the direction it was going in.

I wish Xojo was prouder of their third-party ecosystem. The design award to GraffitiSuite was great and deserved and I’m sure MBS has earned them in the past. At the same time outside of a single session I didn’t see a general awareness for the third-party ecosystem. More plugin and providers should be showcased. While self-serving, I wish there was a certified hosting program. ServerWarp would love to be a certified Xojo web host. Someone at XDC mentioned why there was no section for vendors. I whole heartedly agree.

Everyone believes they attempt to do too much.

I think everyone in the conference was in awe at the tremendous effort and progress of the overall platform. However, it is painfully obvious to us that while engineer staffing may be optimal the overall ecosystem is not healthy. Statistics and marketing data presented to us suggests that the number of women and young people has risen. Yet forum participation and general Xojo developer penetration seems flat or worse. It does not seem entirely impossible that perhaps the number of adult males has simply declined. 

During the final feedback session Geoff was great. He talks about vision and opportunity. Yet, I find myself disgruntled because I am convinced the company lacks technical vision. Nowhere in the conference did we see anything awe inspiring or cutting edge from Xojo. Just further iteration on the same old ideas and concepts with the same limitations and caveats. 

Upon later reflection I realized that Xojo has quietly made some very strong and strategic technical choices. For a long time, they suffered from tremendous “not invented here syndrome” and would recreate everything. Custom database server? Check. Custom HTTP socket? Check. Custom web framework? Check.

Yet more and more these items are being deprecated. Engineering efforts now are on pulling in the best and most accessible libraries available and simplifying them for us. The potential output of any given engineer is likely doubled or tripled due to the reliance on more third-party code. The new web framework for example is relying heavily on bootstrap and jQuery and various other user interface libraries. Interops is designed specifically to make utilizing operating system libraries easier so Xojo itself can iterate faster.

I think the keynote was lackluster because they were unable to communicate how much impact the new technical investments are going to have. As soon as web framework 2.0 is done they will be able to create considerably more controls very quickly. Plus, I hope they are dog-fooding their own Web SDK as much as possible. Android even if terrible will open several new doors and channels to insert Xojo into the conversation. 

So now it is more confusion for me versus disappointment. They apparently do have technical vision but are completely incapable or unwilling to share it. No blogs or forum posts about new abilities or plans. No videos highlighting features being copied from other languages and frameworks. You don’t see the passion and drive that fuels this effort, so it feels hollow. Like a grind. And it becomes one for us constantly following along hoping our feedback request gets fixed and our subscription renewal becomes worthwhile once again.

What do I want?

Monthly alpha builds
Certified third party ecosystem for plugins AND services
Less management of my expectations

I think a little more effort into battling the not invented here syndrome when it comes to community and the third-party market would change the game. If you aspire to be more than just a secret weapon you need to shepherd its growth versus dictating it.

XDC is about being a part of the conversation. Being heard. It should not take a physical gathering to keep that spirit alive.

Freezing Xojo Web Apps

As reported by Ralph Alvy and several others on the forums ( and in Feedback (53291) Xojo web applications built in Xojo 2018r1+ can lock up with no visual indication or recovery.

This is most prevalent on mobile devices when the user is interacting with your Xojo Web app and then moves on to something else. Behind the scenes the iOS or Android operating system suspends the javascript execution of the browser tab. This severs the link between the user application and the Xojo Web server.

In Xojo 2017 and before this was not a problem because the entire communication stack between the browser and server was built on XMLHttpRequest. This is the browser provided class that enables all AJAX interactivity on the web. Since the inception of AJAX however many new technologies like WebSocket’s and Server-sent Events (SSE) have been released.

They provide a persistent connection between the browser and server in either bidirectional or unidirectional capacities. This helps web applications respond faster to changes as building up a request is very expensive in regards to latency and when generating tons of events can really impair web application performance.

In Xojo 2018r1+ the web framework now defaults to SSE as opposed to XMLHttpRequest for all devices (except Firefox). This is great but unfortunately there is a bug. When a users session times out the users browser needs to be reloaded so a new session can be created and use continued. In the old XMLHttpRequest handlers this was recognized and if a session is no longer available on the server then the browser reloads itself.

Sadly, with Xojo’s SSE implementation there is no recognition that the connection to the server has been closed. A generic error has fired but because Xojo Web was working fine prior to the users tab being suspended it is not aware of connectivity issues. The app essentially locks up from the users perspective as the web framework is no longer listening for server side events and client generated events are no longer capable of firing as the session is invalid. Technically the client generated events do fire but the server rejects them.

Fortunately, there is an easy fix! When the SSE connection is closed the ‘readyState’ property of EventSource ( changes to the integer 2. I have verified a fix for this behavior by dynamically changing the error handler of Xojo’s SSE implementation to recognize a ‘readyState’ of 2. When it sees that it recognizes there is no recovery possible and it then starts pinging the server to see if it is still available. If it is then it reloads the browser and starts a new session exactly as the XMLHttpRequest implementation does.

To apply the fix is very simple. You want to execute some Javascript after the framework has fully loaded. Preferably you do this when your first window ‘Shown’ event fires. So do the following:

  1. Add an event handler for the ‘Shown’ event for the first WebPage in your Xojo Web project if it does not already exist.

  2. Add the following code:

    me.ExecuteJavaScript("if (Xojo.eventsource !== undefined){Xojo.eventsource.onerror = function (event) {Xojo.eventsource.msgcount--;if (Xojo.eventsource.msgcount < 1 || Xojo.eventsource.readyState === 2) {Xojo.view.preventInteraction();}};};")

  3. That’s it! You can now deploy your app and verify the fix works on ALL devices.

Feedback for Xojo

  1. There is a lot of parsing in the framework going on for WebSockets, Server-sent events, and conventional AJAX to identify when connections have broken, server unavailable, etc. In most cases you capture these events just fine and do the appropriate thing. However, at times when you miss one the web applications appears “locked up” with no obvious way to resume. You don’t know if it is lagging out, frozen, or what.

    I recommend when the server sees data being sent to a session that is no longer valid that instead of returning a generic 404 you send back a “session invalid error” that can be handled. You can then force the browser to reload regardless of the failures that led to the error in the first place. If the session is invalid there is no point in letting the client linger any longer than necessary as no recovery is possible.

  2. There is a lot of EventSource handling code in Xojo.comm.ajax that relies on message counting to determine if there was connectivity failure. However, if connections break down or fail there is no graceful cutover back to XMLHttpRequest as a fail safe. The current implementation just reloads the browser and then attempts to use SSE again despite a failure in the previous attempt. Instead gracefully switch to XMLHttpRequest and only then if failures persist continue with a reload.

  3. You should revisit the absence of using SSE in Firefox. Mozilla is very standards based and I’m sure appropriate workarounds could be implemented to support this methodology.

  4. This is the second Javascript framework issue I have fixed (45691). My previous fix I submitted to Feedback on June 3, 2017 and all you have to do is implement it! Why the delay? I will submit this resolution to Feedback as well but I hope it can be rolled into R4.

  5. One of the great things about Xojo Web is I can investigate and fix the client-side issues. I hope this provides some evidence that an open source standard library would be a good thing! The bits that make Xojo the most valuable (The IDE, the compiler, etc.) can and should remain closed source. The non-proprietary implementations of standard protocols (SSE) could benefit from more eyes on them in my opinion. See part 2 of my 2017 review series on where I talked about opening up more of Xojo’s standard libraries for cases just like these: