← Bytedance Interview Insights
Started with df to confirm which filesystem, then du to find the culprits.
Structure your answer as a logical troubleshooting narrative: start by confirming the symptom and assessing impact, then systematically narrow down the cause from high-level to specific, and finally resolve and prevent recurrence. Emphasize the reasoning behind each command—what you expect to learn and how it informs the next step—rather than just listing tools.
Pro tip: Before diving into commands, mention that you'd first check if this is a production incident and communicate with stakeholders; this shows you prioritize impact and communication over pure technical debugging. Also, highlight that you'd use 'df -i' early to rule out inode exhaustion, a common but often overlooked cause.
Verify the disk full alert, determine which filesystem is affected, and assess the impact on services and users. Communicate with stakeholders if it's a production issue.
Use tools like df, du, and lsof to find which directories or files are consuming space, and check for deleted-but-open files or inode exhaustion.
Investigate why the space was consumed: logs, core dumps, user uploads, or misconfigured applications. Determine if it's a one-time event or a recurring pattern.
Safely free up space by rotating logs, removing unnecessary files, or truncating large files. Ensure services are restarted if needed and verify disk usage returns to normal.
Implement monitoring, log rotation, quotas, or cleanup scripts to avoid future occurrences. Document the incident and share learnings with the team.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This is where I got a little turned around.
Start by acknowledging the discrepancy between df and du, then systematically investigate common causes such as deleted files held open by processes, filesystem-specific issues (e.g., reserved blocks, inodes), and mount points. Emphasize a methodical troubleshooting process, using tools like lsof, debugfs, and checking /proc for open file descriptors, while considering trade-offs between quick fixes and root cause analysis.
Pro tip: Mention that in production environments, you'd first check for deleted-but-open files using lsof +L1, as this is the most common cause and can be resolved without downtime by restarting the offending process. Also, note that df reports filesystem-level usage while du reports file-level usage, so discrepancies often arise from filesystem overhead or hidden files.
Confirm that df and du are run on the same filesystem and with consistent options (e.g., -h, -x). Check if the issue persists after clearing caches or rebooting, to rule out stale data.
Use lsof +L1 or lsof | grep deleted to find files that have been deleted but are still held open by processes, preventing space from being reclaimed. Restart the processes to free space.
Check for reserved blocks (tune2fs -l), inode exhaustion (df -i), and filesystem metadata overhead. Also consider sparse files, which du may report differently than df.
Check for files in directories that du might skip due to permissions or mount points (e.g., other filesystems mounted under the directory). Use find with -xdev to stay on the same filesystem.
If needed, use debugfs or xfs_db to inspect filesystem internals. Balance quick mitigation (e.g., restarting services) with root cause analysis to prevent recurrence.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the scope (single host vs. cluster) and the symptom (disk full vs. high I/O). Then walk through a systematic diagnostic process: identify the disk, find the top consumers, and trace them to processes using tools like lsof, iotop, and /proc. Emphasize a methodical, tool-agnostic approach that scales from a single server to a distributed system.
Pro tip: Mention that you'd first check if it's a space issue or an I/O issue, as the tools differ (du/df vs. iotop/pidstat). Also, note that in containerized environments, you may need to inspect the host or use cgroup metrics.
Determine whether the issue is disk space exhaustion or high disk I/O, and whether it's on a single machine or across a cluster. This guides tool selection and scope.
Use df -h to find the full disk, then du -sh /* or ncdu to locate the largest directories. For I/O, use iostat or iotop to spot high utilization.
For space, use lsof +L1 to find deleted-but-open files, and lsof /path to see processes using files. For I/O, use iotop -o or pidstat -d to identify processes with high read/write.
Inspect /proc/<pid>/io for per-process I/O counters, and check file descriptors in /proc/<pid>/fd. Correlate with application logs or metrics to confirm the culprit.
Once identified, take action (e.g., restart process, clean logs, adjust retention) and suggest monitoring/alerting to catch similar issues early.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clearly defining inodes and blocks as separate layers of the filesystem: inodes store metadata, blocks store data. Then explain how to check each independently using commands like `df -i` for inodes and `df -h` for blocks, and discuss the implications of exhaustion for each. Finally, tie it back to real-world scenarios like running out of inodes despite free space.
Pro tip: Mention that inode exhaustion can happen even with free disk space, and that it's often caused by many small files; this shows you understand production pitfalls. Also, note that some filesystems (e.g., XFS) allocate inodes dynamically, so the 'inodes vs blocks' trade-off isn't universal.
Explain that inodes are metadata structures storing file attributes and block pointers, while blocks are the actual data storage units. Emphasize that each file consumes one inode and one or more blocks.
Describe using `df -i` to show inode usage per filesystem, and `ls -i` to see a file's inode number. Mention that `stat` can show inode details.
Describe using `df -h` to show block usage in human-readable format, and `du` to check directory sizes. Note that `df` reports filesystem-level block usage.
Explain that running out of inodes prevents new file creation even if blocks are free, and vice versa. Mention that inode count is fixed at filesystem creation (for ext4) and can be a bottleneck for many small files.
Connect to Bytedance's scale: choosing filesystems or designing storage services must consider inode limits, block sizes, and workload patterns (e.g., many small files vs large files).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.