Download Cache Zip LINK
This cache is purely an optimization mechanism. It is completely different to the Conan package cache, (typically the /.conan folder).It is not related to the short_paths mechanism for long path in Windows, nor to the short_paths cache folder. The cache will contain a copyof the artifacts, it is not a new location of files. Those files will still be copied to the Conan package cache, which will not change anything,its behavior, layout or location of any file.
Download Cache zip
The downloads cache implements exclusive locks for concurrency, so it can be shared among different concurrent Conan instances.This is a typical scenario in CI servers, in which each job uses a different Conan package cache (defined by CONAN_USER_HOME environmentvariable). Every different Conan instance could configure its download cache to share the same storage. The download cache implements interprocessexclusive locks, so only 1 process will access at a time to a given cached artifact. If other processes needs the same artifact, they will waituntil it is released, avoiding multiple downloads of the same file, even if they were requested almost simultaneously.
For Continuous Integration processes, it is recommended to have a different Conan package cache (CONAN_USER_HOME) for each job, in most of the cases,because the Conan package cache is not concurrent, and it might also have old dependencies, stale packages, etc. It is better to run CI jobs in a cleanenvironment.
The download cache will store a lot of artifacts, for all recipes, packages, versions and configurations that are used. This can grow and consumea lot of storage. If you are using this feature, provide for a sufficiently large and fast download cache folder.
At the moment, it is only a folder. You can clean the cached artifacts just by removing that folder and its contents. You might also be able torun scripts and jobs that remove old artifacts only. If you do such operations, please make sure that there are not other Conan processes usingit simultaneously, or they might fail.
At the moment I have a NodeJS webapp in the making which scrapes a website for data. Specifically, this webapp scrapes images for the purpose of downloading them. For example, all the image permalinks are scraped from the reddit front page. They are then sent to the client to download individually. My issue is with the website I am scraping there can be thousands of images.
I am relatively new to this entire process and I can't find anything to actually zip the files in the browser cache. I can implement option A relatively easily however this requires a large amount of bandwidth, something I can find for around $5/MO on DigitalOcean. However this entire project is a learning experience and as a result I would love to be able to manage files in the browser cache instead.
If a module might be depended on by other modules, these rules must be followedso that the go command can find and download the module. There are alsoseveral lexical restrictions on characters allowed inmodule paths.
For each entry in the GOPROXY list, the go command requests the latestversion of each module path that might provide the package (that is, each prefixof the package path). For each successfully requested module path, the gocommand will download the module at the latest version and check whether themodule contains the requested package. If one or more modules contain therequested package, the module with the longest path is used. If one or moremodules are found but none contain the requested package, an error isreported. If no modules are found, the go command tries the next entry in theGOPROXY list. If no entries are left, an error is reported.
If the module path appears in a require directive and is not replaced, orif the module paths appears on the right side of a replace directive,the go command may need to download modules with that path, and someadditional requirements must be satisfied.
Most go commands may run in Module-aware mode or GOPATH mode. Inmodule-aware mode, the go command uses go.mod files to find versioneddependencies, and it typically loads packages out of the modulecache, downloading modules if they are missing. In GOPATHmode, the go command ignores modules; it looks in vendordirectories and in GOPATH to find dependencies.
In module-aware mode, GOPATH no longer defines the meaning of imports during abuild, but it still stores downloaded dependencies (in GOPATH/pkg/mod; seeModule cache) and installed commands (in GOPATH/bin, unlessGOBIN is set).
When using modules, the go command typically satisfies dependencies bydownloading modules from their sources into the module cache, then loadingpackages from those downloaded copies. Vendoring may be used to allowinteroperation with older versions of Go, or to ensure that all files used for abuild are stored in a single file tree.
When vendoring is enabled, build commands like go build andgo test load packages from the vendor directory instead of accessing thenetwork or the local module cache. The go list -m command onlyprints information about modules listed in go.mod. go mod commands such asgo mod download and go mod tidy do notwork differently when vendoring is enabled and will still download modules andaccess the module cache. go get also does not work differently whenvendoring is enabled.
The go install command builds and installs the packages named by the pathson the command line. Executables (main packages) are installed to thedirectory named by the GOBIN environment variable, which defaults to$GOPATH/bin or $HOME/go/bin if the GOPATH environment variable is not set.Executables in $GOROOT are installed in $GOROOT/bin or $GOTOOLDIR insteadof $GOBIN. Non-executable packages are built and cached but not installed.
The go mod download command downloads the named modules into the modulecache. Arguments can be module paths or modulepatterns selecting dependencies of the main module or versionqueries of the form path@version. With no arguments,download applies to all dependencies of the main module.
The go command will automatically download modules as needed during ordinaryexecution. The go mod download command is useful mainly for pre-filling themodule cache or for loading data to be served by a moduleproxy.
When vendoring is enabled, the go command will load packages from the vendordirectory instead of downloading modules from their sources into the modulecache and using packages those downloaded copies. See Vendoringfor more information.
go mod verify checks that dependencies of the main modulestored in the module cache have not been modified sincethey were downloaded. To perform this check, go mod verify hashes eachdownloaded module .zip file and extracted directory, thencompares those hashes with a hash recorded when the module was firstdownloaded. go mod verify checks each module in the buildlist (which may be printed with go list -m all).
In contrast, go mod verify checks that module .zip files and their extracteddirectories have hashes that match hashes recorded in the module cache when theywere first downloaded. This is useful for detecting changes to files in themodule cache after a module has been downloaded and verified. go mod verifydoes not download content for modules not in the cache, and it does not usego.sum files to verify module content. However, go mod verify may downloadgo.mod files in order to perform minimal versionselection. It will use go.sum to verify thosefiles, and it may add go.sum entries for missing hashes.
The -modcacherw flag (accepted by go build andother module-aware commands) causes new directories in the module cache tobe writable. To pass -modcacherw to all module-aware commands, add it to theGOFLAGS variable. GOFLAGS may be set in the environment or with go env -w. Forexample, the command below sets it permanently:
The go command caches most content it downloads from module proxies in itsmodule cache in $GOPATH/pkg/mod/cache/download. Even when downloading directlyfrom version control systems, the go command synthesizes explicit info,mod, and zip files and stores them in this directory, the same as if it haddownloaded them directly from a proxy. The cache layout is the same as the proxyURL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) would let users access cached module versions bysetting GOPROXY to
The go command may download module source code and metadata from a moduleproxy. The GOPROXY environmentvariable may be used to configure which proxies thego command may connect to and whether it may communicate directly withversion control systems. Downloaded module data is saved in the modulecache. The go command will only contact a proxy when itneeds information not already in the cache.
When the go command computes the build list, it loads the go.mod file foreach module in the module graph. If a go.mod file is notin the cache, the go command will download it from the proxy using a$module/@v/$version.mod request (where $module is the module path and$version is the version). These requests can be tested with a tool likecurl. For example, the command below downloads the go.mod file forgolang.org/x/mod at version v0.2.0:
In order to load a package, the go command needs the source code for themodule that provides it. Module source code is distributed in .zip files whichare extracted into the module cache. If a module .zip is not in the cache,the go command will download it using a $module/@v/$version.zip request.
Note that .mod and .zip requests are separate, even though go.mod filesare usually contained within .zip files. The go command may need to downloadgo.mod files for many different modules, and .mod files are much smallerthan .zip files. Additionally, if a Go project does not have a go.mod file,the proxy will serve a synthetic go.mod file that only contains a moduledirective. Synthetic go.mod files are generated by thego command when downloading from a version control system.
For example, suppose the go command is attempting to download the moduleexample.com/gopher at version v1.0.0. It sends a request to -get=1. The server responds with an HTML documentcontaining the tag: 041b061a72
