Real-Time Apache Log Streaming with FastAPI

Real-Time Apache Log Streaming with FastAPI

Arjun Adhikari's photo
Arjun Adhikari
·

4 min read

Table of contents

Here, we're going to stream Apache logs by leveraging FastAPI, a high-performance Python framework. FastAPI is known for its speed and ease of use, making it a great choice for building APIs and handling real-time data streams. Let's get started on how to set this up!

Setting Up Your Environment

Before we begin, ensure you have Python installed. FastAPI runs on Python 3.6+.

  1. Create a Virtual Environment (Optional but Recommended): This keeps your project dependencies separate from your global Python installation.

     python -m venv venv
 source venv/bin/activate  # On Windows use `venv\Scripts\activate`

  2. Install FastAPI and Uvicorn: Uvicorn is an ASGI server that will run our FastAPI application.

     pip install fastapi uvicorn

  3. Install Additional Dependencies: We'll use aiofiles for asynchronous file handling.

     pip install aiofiles

Building the FastAPI Backend

The FastAPI App

We will create a file namedmain.py and write the FastAPI application code such that it sends new content from the Apache logs when it's added. We also need to implement a way to watch the file for changes and send those changes over the WebSocket. This can be achieved using file polling or more sophisticated file monitoring techniques. However, keep in mind that Python doesn't have built-in, cross-platform support for real-time file monitoring, so we'll use a simple polling approach for this example.

FastAPI Backend with File Polling

Here's how you can adjust the main.py to include file polling:

from fastapi import FastAPI, WebSocket
from fastapi.responses import HTMLResponse
import aiofiles
import asyncio
import os

app = FastAPI()

html = """
<!DOCTYPE html>
<html>
    <head>
        <title>Apache Logs Stream</title>
    </head>
    <body>
        <h1>Apache Access Log</h1>
        <pre id="accessLog"></pre>
        <script src="/static/app.js"></script>
    </body>
</html>
"""

@app.get("/")
async def get():
    return HTMLResponse(html)

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    file_path = '/path/to/apache.access.log'
    last_size = 0

    while True:
        await asyncio.sleep(1)  # Poll every second
        current_size = os.path.getsize(file_path)
        if current_size > last_size:
            async with aiofiles.open(file_path, mode='r') as f:
                await f.seek(last_size)
                while new_line := await f.readline():
                    # If new line is just a blank line, new line char or empty string, skip it
                    if new_line in ["\n", "\r\n", ""]:
                        continue
                    await websocket.send_text(new_line)
            last_size = current_size

In this code:

  1. The WebSocket connection is established.

  2. It enters an infinite loop where it checks the file size every second.

  3. If the file size has increased (indicating new content), it reads the new lines and sends them over the WebSocket.

Remember to replace /path/to/apache.access.log with the actual path to your Apache access log file.

Running the Application

Run your FastAPI server as before:

uvicorn main:app --reload

Now, when you add new lines to your Apache log file, those lines will be sent to the connected web client in real-time.

Important Considerations

  • Polling vs. Event-Driven Monitoring: This example uses polling, which is less efficient than event-driven monitoring. For a production environment, you might want to explore libraries like watchdog for Python, which can provide more efficient file monitoring.

  • Error Handling: In a production environment, add error handling to deal with issues like lost WebSocket connections or file read errors.

  • Security and Performance: Be cautious about security implications and the performance impact of constantly reading large log files, especially in a busy production environment.Replace /path/to/apache.access.log with the path to your Apache log file.

The Frontend JavaScript

In a folder named static, create app.js with the following content:

const ws = new WebSocket('ws://localhost:8000/ws');
ws.onmessage = function(event) {
    document.getElementById('accessLog').textContent += event.data + '\n';
};

Key Takeaways

  • Asynchronous Streaming: FastAPI and aiofiles make it easy to handle real-time data streaming asynchronously.

  • Security: Remember to implement authentication and secure your WebSocket connections, especially in a production environment.

  • Performance: FastAPI is known for its high performance, making it an excellent choice for real-time data streaming applications.

Conclusion

Using FastAPI for streaming Apache logs to a web interface is a powerful approach. It's not only efficient but also provides a modern Pythonic way of handling real-time data. This setup can be customized to fit a variety of monitoring and streaming needs.

Happy coding and real-time monitoring with FastAPI! 🐍💨

FastAPIPythonapachestreaming