Machine learning operations (MLOps) refer to the emerging field of delivering machine learning models through repeatable and efficient workflows. The machine learning lifecycle is composed of various elements, as shown in the figure below. Similar to the practice of DevOps for managing the software development lifecycle, MLOps enables organizations to smooth the path to successful AI transformation by providing an engineering and technological backbone to underlying machine learning processes.
MLOps is a relatively new field, as the commercial use of AI at scale is itself a fairly new practice. MLOps is modeled on the existing field of DevOps, but in addition to code, it incorporates additional components, such as data, algorithms, and models. It includes various capabilities that allow the modern machine learning team, comprising data scientists, machine learning engineers, and software engineers, to organize the building blocks of machine learning systems and take models to production in an efficient, reliable, and reproducible fashion.
MLOps is carried out using a diverse set of tools, each catering to a distinct component of the machine learning pipeline. Each tool under the MLOps umbrella is focused on automation and enabling repeatable workflows at scale. As the field of machine learning has evolved over the last decade, organizations are increasingly looking for tools and technologies that can help extract the maximum return from their investment in AI. In addition to cloud providers, like AWS, Azure, and GCP, there are a plethora of start-ups that focus on accommodating varied MLOps use cases.
In this article, I will cover tools for the following MLOps categories:
In the following section, I will list a selection of MLOps tools from the above categories. It is important to note that although a particular tool might be listed under a specific category, the majority of these tools have evolved from their initial use case into a platform for providing multiple MLOps solutions across the entire ML lifecycle.
Building machine learning models involves many parameters associated with code, data, metrics, model hyperparameters, A/B testing, and model artifacts, among others. Reproducing the entire ML workflow requires careful storage and management of the above metadata.
Featureform is a virtual feature store. It can integrate with various data platforms, and it enables the management and governance of the data from which features are built. With a unique, feature-first approach, Featureform has built a product called Embeddinghub, which is a vector database for machine learning embeddings. Embeddings are high-dimensional representations of different kinds of data and their interrelationships, such as user or text embeddings, that quantify the semantic similarity between items.
MLflow is an open-source platform for the machine learning lifecycle that covers experimentation and deployment, and it also includes a central model registry. It has four principal components: Tracking, Projects, Models, and Model Registry. In terms of metadata management, the MLflow Tracking API is used for logging parameters, code, metrics, and model artifacts.
For machine learning systems, versioning is a critical feature. As the pipeline consists of various data sets, labels, experiments, models, and hyperparameters, it is necessary to version control each of these parameters for greater accessibility, reproducibility, and collaboration across teams.
Pachyderm provides a data layer for the machine learning lifecycle. It offers a suite of services for data versioning that are organized by data repository, commit, branch, file, and provenance. Data provenance captures the unique relationships between the various artifacts, like commits, branches, and repositories.
DVC, or Data Version Control, is an open-source version control system for machine learning projects. It includes version control for machine learning data sets, models, and any intermediate files. It also provides code and data provenance to allow for end-to-end tracking of the evolution of each machine learning model, which promotes better reproducibility and usage during the experimentation phase.
A typical machine learning system may only be deployed after hundreds of experiments. To optimize the model performance, data scientists perform numerous experiments to identify the most appropriate set of data and model parameters for the success criteria. Managing these experiments is paramount for staying on top of the data science modeling efforts of individual practitioners, as well as the entire data science team.
Comet is a machine learning platform for managing and optimizing the entire machine learning lifecycle, from experiment tracking to model monitoring. Comet streamlines the experimentation workflow for data scientists and enables clear tracking and visualization of the results of each experiment. It also allows side-by-side comparisons of experiments so users can easily see how model performance is affected.
Weights & Biases
Weights & Biases is another popular machine learning platform that provides a host of services, including [experiment tracking](https://wandb.ai/site/experiment-tracking). It facilitates tracking and visualization of every experiment, allows rerunning previous model checkpoints, and can monitor CPU and GPU usage in real time.
Once a machine learning model is built and tests have found it to be robust and accurate enough to go to production, the model is deployed. This is an extremely important aspect of the machine learning lifecycle, and if not managed well, it can lead to errors and poor performance in production. AI models are increasingly being deployed across a range of platforms, from on-premises servers to the cloud to edge devices. Balancing the trade-offs for each kind of deployment and scaling the service up or down during critical periods are very difficult tasks to achieve manually. A number of platforms provide model deployment capabilities that automate the entire process of taking a model to production.
Seldon is a model deployment software that helps enterprises manage, serve, and scale machine learning models in any language or framework on Kubernetes. It’s focused on expediting the process to take a model from proof of concept to production, and it’s compatible with a variety of cloud providers.
Kubeflow is an open-source system for productionizing models on the Kubernetes platform. It simplifies machine learning workflows on Kubernetes and provides greater portability and scalability. It can run on any hardware and infrastructure on which Kubernetes is running, and it is a very popular choice for machine learning engineers when deploying models.
Once a model is in production, it is essential to monitor its performance and log any errors or issues that may have caused the model to break in production. Monitoring solutions enable setting thresholds as indicators for robust model performance and are critical in solving for known issues, like data drift. These tools can also monitor the model predictions for bias and explainability.
Fiddler is a machine learning model performance monitoring software. To ensure expected model performance, it monitors data drift, data integrity, and anomalies in the data. Additionally, it provides model explainability solutions that help identify, troubleshoot, and understand underlying problems and causes of poor performance.
Evidently is an open-source machine learning model monitoring solution. It measures model health, data drift, target drift, data integrity, and feature correlations to provide a holistic view of model performance.
MLOps is a growing field that focuses on organizing and accelerating the entire machine learning lifecycle through best practices, tools, and frameworks borrowed from the DevOps philosophy of software development lifecycle management. With machine learning, the need for tooling is much greater, as machine learning is built on foundational blocks of data and models, as well as code.
To bring reliability, maturity, and scale to machine learning processes, a diverse set of MLOps tools are being increasingly used. These tools are developed for optimizing the nuts and bolts of machine learning operations, including metadata management, versioning, model building and experiment tracking, model deployment, and monitoring in production.
Over the past decade, the field of AI and machine learning has grown rapidly, with organizations embracing AI and recognizing its critical importance for transforming their business. The field of MLOps is still young, but the creation and adoption of tools will further empower organizations in their journey of AI transformation and value creation.
Consumer technology companies like Amazon, Yelp, and Airbnb are focused on providing an impeccable customer experience, and reviews are integral to that experience. Reviews from previous customers can signal trust and reliability (e.g., total number of reviews or average star rating), empowering first-time buyers or new customers in their decision-making. Millions of reviews are shared on platforms like Amazon for e-commerce products, on Airbnb for travel and hospitality, on Glassdoor for company and employment experience, and on Google for third-party businesses.
However, the internet has become rife with fake reviews. Fake reviews and inflated ratings provide a tainted picture of a product or service and are designed to trick customers away from or toward certain purchases. As these reviews are an important input factor for search and ranking algorithms, they can have a massive influence on product discovery and sales. This provides a strong incentive for bad actors to try to manipulate the system by improving the ratings of their products through fake reviews.
There is a booming market for fake reviews, which are purchased via multiple social media and community platforms. The problem is enormous - nearly four percent of all reviews are fake, translating into a global economic impact of USD 152 billion.
E-commerce companies like Amazon spend upwards of a billion dollars and employ tens of thousands of workers to combat online fraud and abuse. Some companies use sophisticated technologies including AI to detect and delete fake reviews, but their accuracy is limited (less than forty percent) and it often takes more than one hundred days to remove those reviews. During that time, fraudulent sellers can make strong short-term revenues and profits.
Apart from the short-term commercial losses, there is a longer-term problem; fake reviews erode customer trust and safety, causing customers to avoid online purchases. Catching fake reviews is therefore paramount for a majority of online marketplaces and businesses.
Characteristics of Fake Reviews
Fake reviews have several telltale characteristics. For instance, as they are based on a fraudulent experience with the product or service, fake reviews will often focus on a poor customer experience without specific details about that product or service.
Another sign is the repetition of positive or negative keywords and text. As it is difficult to fabricate a review, fake reviewers keep emphasizing certain keywords and details to paint a terrible customer experience. Such reviews accentuate extreme details without providing a balanced perspective.
Fake reviewers also excessively use emoticons and exclamation points in an attempt to appeal to the customers’ emotions. Genuine reviewers tend to focus more on information and provide thoughtful, actionable feedback for other customers about the product experience.
One clear giveaway is the reviewer’s name and avatar. Fake reviews are usually submitted using an account with a dubious username, avatar, or email address. If a reviewer seems like they could be illegitimate, check whether they have shared any reviews previously, how often, and for which products or businesses. Fake accounts are often created for one-time use, and fake reviewers can submit multiple reviews in a short span of time, sometimes on the same day.
Sometimes fake reviewers post a poor rating without any comments to describe their experience. Genuine reviewers take the time and effort to write useful feedback.
Spotting these characteristics can help you find many fake reviews, but scamsters are always devising more sophisticated techniques to replace those that have already been detected through algorithms, AI, or human reviewers.
One solution to this problem is fingerprinting technology, that can identify unique users of your website regardless of VPNs, cookie blockers, private browsing, or other tools. They use data including the browser and device used, usage patterns, IP addresses, and geolocation to create a unique identifier for site visitors, making it easier to spot users trying to hide their identity or committing fraudulent activity.
Fake reviews have undermined the revenue and growth of online sellers and small businesses. These reviews can boost the sales of a poor product by exaggerating its positive rating, or damage the sales of competitor products via negative reviews. While there are ways to catch these fake reviewers in the act, it’s an increasingly sophisticated scam and a headache for businesses.
Fingerprinting technology can help you find and remove fake reviews as well as protect your business from all types of online fraud. This helps ensure that your customers will have a safe and reliable online shopping experience.
Recently, the Government of India issued a draft framework of standards to counter fake reviews in order to reduce their prevalence on e-commerce platforms.
If you have worked on a large engineering product, the odds are that you did not build it in isolation but in collaboration with colleagues from diverse domains. It’s common for software projects to involve product, business, marketing, design, analytics, customer support, end-users, and C-suite leaders. All of these people with a vested interest in the success of your product represent your stakeholders.
The tech industry is littered with numerous failed products from Amazon’s Fire phone to Google+. One of the common recurring reasons is poor stakeholder management and coordination.
Knowing who your stakeholders are, understanding their incentives and priorities, aligning their interests, getting buy-in, and leveraging their expertise is critical for the success of your product. Just one of the stakeholders being misaligned makes developing your product much harder, so it is imperative that you manage stakeholders from the conception to the launch of your product.
I have seen several products fail due to poor planning and management of stakeholders. In this article, I will provide some insights on effectively managing your stakeholders as an engineering leader.
Identifying Your Stakeholders
The first step is to identify the stakeholders that are critical to the success of your product, understand their motivation and priorities, and their relative importance and influence. Successful stakeholder management starts by mapping your stakeholders across several dimensions, including:
A stakeholder mapping exercise will help you identify the most important stakeholders during each phase of the product development lifecycle. This helps you later as you develop optimal strategies to balance the diverse perspectives of each stakeholder, manage any inevitable conflicts, and build unique communication methods for each one.
An important point that many articles on this topic omit is that stakeholder mapping is not static. It must be modified as individual stakeholders are replaced or reallocated to the project. For instance, if your VP of Product is replaced by a new one, the mapping has to be redrawn based on your new VP’s motivations and priorities. These will almost certainly affect your roadmap for any products under development.
Prioritizing Your Stakeholders
Every stakeholder contributes unique expertise that is pivotal for your product’s success, but their relative interest, importance, influence, and level of contribution varies across the product lifecycle. Using your stakeholder map, identify the most important stakeholders and engage them accordingly.
Remember that CXOs aren’t necessarily the most important stakeholders at every stage of a product’s development. For instance, while C-level business leaders are crucial for providing initial budget and sign-off, during the development phase, your product and design stakeholders are often more relevant.
Furthermore, if you lose touch with the end-users during the development phase, you might end up building a product missing features that customers want. I saw this recently when another engineering team started a new project.
After getting initial buy-in from leadership and validating the customers’ needs, the manager was replaced. Instead of re-assessing the team’s priorities and confirming stakeholder alignment, the team got out of sync with the deliverables that customers needed.
At the end of a year-long development cycle, they launched the product, but customers were no longer bought in. The team essentially lost a year’s worth of effort because they didn’t keep all the relevant stakeholders in mind throughout the development process.
While stakeholder mapping is a necessary first step, conflicting priorities amongst stakeholders are common and need to be resolved delicately.
You can preempt certain stakeholder conflicts by keeping conflicting interests in mind. Achieving multi-stakeholder alignment requires carefully planned discussions and negotiations to assess the lay of the land with each stakeholder. Focused group meetings prioritizing key points of disagreement or conflicting priorities can help you achieve alignment and avoid conflicts down the line.
Additionally, knowing which stakeholders are most likely to back your project through challenging times and which are likely to abandon you is important. Learn when to fight fires on your own and when to seek out more experienced mentors in your company to provide a different perspective.
The strategy for stakeholder alignment works differently in small startups than it does in large, multi-layered organizations. In a startup, speed of execution is critical while enterprise teams may have many layers of bureaucracy impeding velocity. Both have their challenges, but it’s important not to treat them the same.
Keeping Stakeholders Up to Date
After getting all the stakeholders aligned, you need a communication strategy to share updates on the project at set intervals. Tailor your communication plan according to the importance, expertise, and priorities of each stakeholder. For example, the project’s contributors might need a high-touch approach, while executives might just want periodic updates and high-level presentations.
There are several appropriate modes of communication apart from 1:1 meetings, especially now that so many teams are remote. At a minimum, you should maintain a project status board detailing the progress of each milestone, KPI, team member, and estimated timeline.
Popular software and tools that facilitate structured project management include:
While a project board might be enough for some stakeholders, others may want more explicit communication. As an engineering leader, be proactive about this communication, leveraging video conferencing and messaging tools to help keep stakeholders in the loop.
Finally, group status updates can be productive when addressing stakeholders who are too busy to read or watch pre-recorded videos. These meetings are especially helpful if you need signoff from several parties to proceed.
Engaging Stakeholders and Taking Feedback
During the execution phase of the project, continuous engagement and clear communication with your stakeholders is essential to maintain their interest and trust. Stakeholders are often involved in a multitude of projects, and your project may not be their top priority.
While giving stakeholders regular updates can help, it’s also helpful to get their buy-in or actively involve them in some parts of the project. As long as the key stakeholders are satisfied, there is less likelihood of difficult surprises down the road.
Often, stakeholders will express disappointment or frustration with your team’s work, rate of progress, or specific decisions related to product development. Feedback - both positive and negative - helps your team apprise their project status and course-correct accordingly, so first, evaluate the feedback to understand whether it’s valid or a simple miscommunication.
Leveraging your objective analysis will enable you to respond to the stakeholders’ concerns accordingly and manage their expectations better in the future. Whenever situations like this arise, follow up by bringing these stakeholders into decisions earlier and ensuring they have the information they need to suggest changes earlier.
Finally, use feedback as an opportunity to understand what contributes to stakeholder satisfaction with the project. There might be a new process you can implement to avoid similar misunderstandings on future projects.
Developing Your Stakeholder Management Skills
For new engineering leaders, the art of stakeholder management might take some time to master. If you feel like you’re struggling with it, find mentors (typically senior engineering leaders in your organization) and look into external professional training. Successful stakeholder management requires a host of soft skills, including communication, empathy, and persuasiveness.
“You need to be able to communicate and collaborate with other engineers who are working on different pieces of that system, as well as cross-functionally with product managers, designers, and others who are contributing to the design of the system that you’ll help implement.” - David Kaminsky, Senior Engineering Manager at StubHub
While some technical experts might dismiss the relevance of these skills when they’re in individual contributor roles, in my experience, the most successful engineering leaders combine strong technical acumen with excellent interpersonal skills.
The success of your software or engineering product depends on your ability to successfully collaborate with and manage cross-functional stakeholders across the entire product development lifecycle.
Identifying your key stakeholders and understanding their priorities lays the foundation for achieving strong cross-functional stakeholder alignment. This foundation can be further cemented with smart communication strategies and tools to keep your stakeholders abreast of the progress and vested in the success of your product.
Stakeholder management is not a new topic, but no two managers do it the same way. You have to incorporate your context, industry, and strengths to find a set of practices that work for you.
Copyright © 2022, Sundeep Teki
All rights reserved. No part of these articles may be reproduced, distributed, or transmitted in any form or by any means, including electronic or mechanical methods, without the prior written permission of the author.
This is a personal blog. Any views or opinions represented in this blog are personal and belong solely to the blog owner and do not represent those of people, institutions or organizations that the owner may or may not be associated with in professional or personal capacity, unless explicitly stated.