How to Manage Large Log Files in Go: Truncate a Log File to a Specific Size

When working with applications, log files can quickly grow to unmanageable sizes, consuming valuable disk space and slowing down systems. To address this, you might want to monitor your log file size and truncate it when it exceeds a specific limit. This blog walks you through implementing such functionality in Go.

Objective

The goal is to create a Go program that:

• Monitors the size of a log file.

• Truncates the file to a specified size (e.g., 500MB) if it exceeds a threshold (e.g., 1GB).

Step 1: Define the Thresholds

We'll define two constants for our file size limits:

• 1GB as the maximum allowed size.

• 500MB as the target size after truncation.

Here’s how to define these values in bytes in Go:

const (
    maxSize = 1 * 1024 * 1024 * 1024 // 1GB in bytes
    truncateSize = 500 * 1024 * 1024 // 500MB in bytes
)

Step 2: Check File Size

Use Go's os.Stat() function to retrieve the size of the file:

fileInfo, err := os.Stat(logFilePath)
if err != nil {
    fmt.Printf("Error accessing file: %v\n", err)
    os.Exit(1)
}
fileSize := fileInfo.Size()
This retrieves the file's metadata, including its size in bytes.

Step 3: Truncate the File

If the file exceeds the size threshold, we need to truncate it to 500MB. Since we want to retain the most recent logs, we’ll:

1. Count the total number of lines in the file.

2. Calculate how many lines correspond to 500MB.

3. Keep only the last targetLines lines.

Here’s the code to achieve this:

// Open the log file for reading
file, err := os.Open(logFilePath)
if err != nil {
    fmt.Printf("Error opening file: %v\n", err)
    os.Exit(1)
}
defer file.Close()
// Count total lines in the file
scanner := bufio.NewScanner(file)
totalLines := 0
for scanner.Scan() {
    totalLines++
}
if err := scanner.Err(); err != nil {
    fmt.Printf("Error reading file: %v\n", err)
    os.Exit(1)
}
// Calculate the number of lines to keep for 500MB
targetLines := int64(totalLines) * truncateSize / fileSize
// Read the last `targetLines` lines
file.Seek(0, 0)
linesToKeep := make([]string, 0, targetLines)
scanner = bufio.NewScanner(file)
lineCount := 0
for scanner.Scan() {
    lineCount++
    if lineCount > int(totalLines)-int(targetLines) {
        linesToKeep = append(linesToKeep, scanner.Text())
    }
}

Step 4: Write to a Temporary File

Once you’ve identified the lines to keep, write them to a temporary file:

tempFilePath := logFilePath + ".tmp"
tempFile, err := os.Create(tempFilePath)
if err != nil {
    fmt.Printf("Error creating temp file: %v\n", err)
    os.Exit(1)
}
defer tempFile.Close()
for _, line := range linesToKeep {
    tempFile.WriteString(line + "\n")
}

Step 5: Replace the Original File

Finally, replace the original file with the truncated file:

if err := os.Rename(tempFilePath, logFilePath); err != nil {
    fmt.Printf("Error replacing file: %v\n", err)
    os.Exit(1)
}

Full Code Example

Here’s the complete program:

package main
import (
    "bufio"
    "fmt"
    "os"
    "strconv"
)
const (
    maxSize      = 1 * 1024 * 1024 * 1024 // 1GB in bytes
    truncateSize = 500 * 1024 * 1024      // 500MB in bytes
)
func main() {
    if len(os.Args) < 2 {
        fmt.Println("Usage: truncate_log /path/to/logfile.log")
        os.Exit(1)
    }
    logFilePath := os.Args[1]
    fileInfo, err := os.Stat(logFilePath)
    if err != nil {
        fmt.Printf("Error accessing file: %v\n", err)
        os.Exit(1)
    }
    fileSize := fileInfo.Size()
    if fileSize >= maxSize {
        fmt.Printf("Log file is %d bytes, truncating to %d bytes.\n", fileSize, truncateSize)
        file, err := os.Open(logFilePath)
        if err != nil {
            fmt.Printf("Error opening file: %v\n", err)
            os.Exit(1)
        }
        defer file.Close()
        scanner := bufio.NewScanner(file)
        totalLines := 0
        for scanner.Scan() {
            totalLines++
        }
        targetLines := int64(totalLines) * truncateSize / fileSize
        file.Seek(0, 0)
        linesToKeep := make([]string, 0, targetLines)
        scanner = bufio.NewScanner(file)
        lineCount := 0
        for scanner.Scan() {
            lineCount++
            if lineCount > int(totalLines)-int(targetLines) {
                linesToKeep = append(linesToKeep, scanner.Text())
            }
        }
        tempFilePath := logFilePath + ".tmp"
        tempFile, err := os.Create(tempFilePath)
        if err != nil {
            fmt.Printf("Error creating temp file: %v\n", err)
            os.Exit(1)
        }
        defer tempFile.Close()
        for _, line := range linesToKeep {
            tempFile.WriteString(line + "\n")
        }
        if err := os.Rename(tempFilePath, logFilePath); err != nil {
            fmt.Printf("Error replacing file: %v\n", err)
            os.Exit(1)
        }
        fmt.Printf("Log file truncated to %d bytes.\n", truncateSize)
    } else {
        fmt.Printf("Log file size is within limits: %d bytes.\n", fileSize)
    }
}

How to Run

1. Save the file as truncate_log.go.

2. Build the program: go build -o truncate_log truncate_log.go

3. Run the program with a log file as an argument: ./truncate_log /path/to/logfile.log

Final Thoughts

This Go program efficiently manages large log files by monitoring their size and truncating them to a manageable size. You can integrate it into your automation pipeline or use it as a standalone tool to keep log files under control.

Have questions or suggestions? Feel free to share them in the comments below!