- Use flags to configure the worker pool and the maximum number of pages. - Add README.md
Add the source code for the web crawler. The web crawler is a simple Go CLI application that traverses through a website and generates a report of all the internal links found in the site.