In a world increasingly dominated by digital advancements, the scope of technology continues to expand rapidly, giving rise to an escalating demand for diverse data to fulfill fundamental informational needs held by authorized organizations and governments. However, a critical challenge arises in this context – the reliance on third-party entities for data provisioning. The inherent issue lies in the dynamic nature of data, which undergoes constant changes over time.
Why is this a concern?
The actions and events executed within our systems often hinge upon the availability and accuracy of this data. The consistency of data across the system becomes paramount for the seamless completion of an event. Enter webhooks, a solution that addresses this very challenge.
In this blog, we delve into the technical aspects of webhooks, exploring how they offer a resolution to the data consistency dilemma.
Webhook?
A web-based event notification system where an application or service can send data to a designated URL or endpoint as soon as a specific event occurs.
A method of communication where one system (the sender) informs another system (the receiver) about events in real-time by making HTTP requests
e.g. The workflow between Jenkins and Git, wherein the initiation of a project is triggered by subscribed events, heavily relies on the webhook mechanism.
"Let's gain insight by implementing a workflow akin to Jenkins and Git, illustrating a mechanism that exemplifies the interaction between the two systems."
To initiate the process, the first step is to set up two servers—one serving as a Jenkins server and the other as a Git server responsible for capturing events and initiating the webhook to notify the designated system, in this case, our Jenkins server.
Now that we have a clear roadmap, let's proceed with the implementation.
Jenkins server, Git Server - Node Express.js.
Jenkins client - react.
Jenkins server
A endpoint that gives list of projects.
A endpoint that will update the project history list based on the project id.
let projects : An array named projects holds information about different projects. Each project has an id, name, and a history array, which captures the project's historical events. ( instead of using an external database here we store it in the memory).This data pertains solely to the Jenkins server.
"/projects" : This endpoint handles HTTP GET requests to "/projects" and responds with the array of projects in JSON format. If an error occurs, it sends a 500 Internal Server Error response.
"/webhook" : This endpoint handles HTTP POST requests to "/webhook". It expects a JSON payload with a projectId and other details. It finds the corresponding project in the projects array, adds the event details to its history, and emits a WebSocket event named "trigger" to the associated project's socket ( jenkins client )which updates the client side in real time. Finally, it responds with a JSON object containing a success message. If an error occurs, it sends a 500 Internal Server Error response.
Jenkins Client:
This will have two screens
/projects: This will list all projects.
/project/:id: This will show the history of the selected project.
Projects screen:
The apiState state manages the loading status of the API request.
const { projects, setProjects } = useProjectstate(); This context is responsible for managing the state related to projects in the application.
The useEffect hook is used to make an asynchronous GET request to the "http://localhost:4000/projects" ( jenkins server endpoint) endpoint. Upon success, it updates the state( project context ) with the fetched project data and sets the loading status to false. Errors during the API request are also handled.
The component renders a loading message if the API is still loading. Once loaded, it maps through the projects array and renders a clickable div for each project. Clicking on a project item navigates to the corresponding project page.
navigate(`project/${project.id}`): React Router's useNavigate hook is used to navigate to the project page when a project item is clicked.
Selected Project page:
Utilizing the useParams hook from React Router to extract the id parameter from the route.
Uses context hooks (useGlobalstate and useProjectstate) to access the WebSocket connection (ws), project state management functions (setProjects and getProject), and global state.
In the useEffect block, a WebSocket connection is initialized by triggering a "join" event with the current project's ID. The component subscribes to "trigger" events, responding to updates from the WebSocket. This mechanism aligns with the earlier observation in the Jenkins server, where a webhook endpoint, upon updating a project's history associated with a specific project ID, emits a WebSocket event labeled "trigger." This event serves to inform connected clients in real-time, ensuring synchronous updates across clients and maintaining real-time synchronization of project details.
Renders a list of historical events for the selected project. Uses the map function to iterate through the project's history and display each event in a styled div.
After comprehending and setting up our Jenkins server and clients, the next step is to execute our application and witness the display of our projects.
Here is a screenshot capturing the history of the selected project page, responsible for showcasing real-time event updates for the project.
(assume some events are being occurred in the mean time)
After patiently waiting for hours to witness real-time project updates, no changes are apparent in the project history, despite events occurring in the source code management server, typically Git.
What went wrong?
The events are indeed being captured on the server, typically the Git server, where the actual project is stored and maintained.
How to solve?
To ensure that events occurring on the Git server trigger corresponding actions on our Jenkins server, we leverage the endpoint "/webhook" that we've previously established.
Overview of the solution:
Creating our own Git server, we can now delve into the implementation details of this solution. This involves configuring the Git server to interact with the "/webhook" endpoint, facilitating seamless communication between Git events and Jenkins actions.
Git Server:
To simplify the process, we establish a single endpoint responsible for triggering the Jenkins endpoint. This, in turn, initiates the update of project details and subsequently notifies the connected clients.
Defines a POST endpoint at "/event" that receives incoming data from the request body.
This data is then forwarded to another endpoint at "http://localhost:4000/webhook" using an Axios POST request.
Through the usage of the /event endpoint on the Git server, any events transpiring in the Git server will activate the Jenkins endpoint. This operational mechanism, where one system (the sender) informs another system (the receiver) of real-time events through HTTP requests, is recognized as the webhook mechanism.
The address furnished by Jenkins for this purpose is termed the webhook URL (http://localhost:4000/webhook).
lets us now observe the project history page in the below video.
Jenkins Server running on 4000, Jenkins Client running on 3000, Git server running on 5000.
As I simulate the event occurrence, I accomplish this by sending a request to the /event endpoint on the Git server using Postman tool.
From the above video, it's evident that with each hit to the /event endpoint, a POST request is made to the /webhook endpoint on the Jenkins server using the webhook URL. Consequently, this action leads to updates in the Jenkins client.
In real-time, the webhook URL is configured specifically for each project in the Source Code Management (SCM) server. This demonstration provides an overview of the entire process, offering an understanding of what a webhook is. The complete mechanism involves more complex steps, and there are additional considerations for error handling.
A diagramatic representation of the Webhook flow
Conclusion:
In conclusion, webhooks represent a powerful and intricate means of communication between systems. Their application is most effective when tailored to specific requirements. While they serve well for real-time scenarios, they may not be the optimal choice for certain systems, such as stock apps, where technologies like websockets are better suited. It is crucial to recognize that the utilization of webhooks is not a universal necessity. Their adoption should be driven by specific needs and a tangible impact on system functionality.