Dave's PCF WIP: Paragraphs

Paragraphs

Actions	Application	Content	Paragraph Number	Notes	Modified
		Content		Paragraph Number	Notes	Any Field
View Edit Delete	2380.2.01	FIG. 4A is a schematic block diagram illustrating one embodiment 400 of a bank interleave controller 344 in the solid-state storage controller 104 in accordance with the present invention. The bank interleave controller 344 is connected to the control and status registers 340 and to the storage I/O bus 210 and storage control bus 212 through the MUX 350, storage bus controller 348, and synchronization buffer 346, which are described below. The bank interleave controller 344 includes a read agent 402, a write agent 404, an erase agent 406, a management agent 408, read queues 410a-n, write queues 412a-n, erase queues 414a-n, and management queues 416a-n for the banks 214 in the solid-state storage 110, bank controllers 418a-n, a bus arbiter 420, and a status MUX 422, which are described below. The storage bus controller 348 includes a mapping module 424 with a remapping module 430, a status capture module 426, and a NAND bus controller 428, which are described below.	173	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	Bank Interleave	172	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	Typically the solid-state storage controller(s) 104 communicate data to the solid-state storage 110 over a storage I/O bus 210. In a typical embodiment where the solid-state storage is arranged in banks 214 and each bank 214 includes multiple storage elements 216, 218, 220 accessed in parallel, the storage I/O bus 210 is an array of busses, one for each row of storage elements 216, 218, 220 spanning the banks 214. As used herein, the term “storage I/O bus” may refer to one storage I/O bus 210 or an array of data independent busses 204. In a preferred embodiment, each storage I/O bus 210 accessing a row of storage elements (e.g. 216a, 218a, 220a) may include a logical-to-physical mapping for storage divisions (e.g. erase blocks) accessed in a row of storage elements 216a, 218a, 220a. This mapping allows a logical address mapped to a physical address of a storage division to be remapped to a different storage division if the first storage division fails, partially fails, is inaccessible, or has some other problem. Remapping is explained further in relation to the remapping module 430 of FIGS. 4A and 4B.	110	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In one embodiment, the object index is stored in memory 230, 232 and then periodically off-loaded to a channel of the solid-state storage 110n or other non-volatile memory. One of skill in the art will recognize other uses and configurations of the memory controller 228, dynamic memory array 230, and static memory array 232.	125	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In another embodiment, the master controller 224 coordinates with an autonomous integrated management controller to periodically validate FPGA code and/or controller software, validate FPGA code while running (reset) and/or validate controller software during power on (reset), support external reset requests, support reset requests due to watchdog timeouts, and support voltage, current, power, temperature, and other environmental measurements and setting of threshold interrupts. In another embodiment, the master controller 224 manages garbage collection to free erase blocks for reuse. In another embodiment, the master controller 224 manages wear leveling. In another embodiment, the master controller 224 allows the data storage device/solid-state storage device 102 to be partitioned into multiple virtual devices and allows partition-based media encryption. In yet another embodiment, the master controller 224 supports a solid-state storage controller 104 with advanced, multi-bit ECC correction. One of skill in the art will recognize other features and functions of a master controller 224 in a storage controller 152, or more specifically in a solid-state storage device 102.	123	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In one embodiment, the master controller 224 is also a redundant array of independent drive (“RAID”) controller. Where the data storage device/solid-state storage device 102 is networked with one or more other data storage devices/solid-state storage devices 102, the master controller 224 may be a RAID controller for single tier RAID, multi-tier RAID, progressive RAID, etc. The master controller 224 also allows some objects to be stored in a RAID array and other objects to be stored without RAID. In another embodiment, the master controller 224 may be a distributed RAID controller element. In another embodiment, the master controller 224 may comprise many RAID, distributed RAID, and other functions as described elsewhere.	120	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In one embodiment, the master controller 224, which manages objects, emulates block storage such that a computer 112 or other device connected to the storage device/solid-state storage device 102 views the storage device 152/solid-state storage device 102 as a block storage device and sends data to specific physical addresses in the storage device 152/solid-state storage device 102. The master controller 224 then divides up the blocks and stores the data blocks as it would objects. The master controller 224 then maps the blocks and physical address sent with the block to the actual locations determined by the master controller 224. The mapping is stored in the object index. Typically, for block emulation, a block device application program interface (“API”) is provided in a driver in the computer 112, client 114, or other device wishing to use the storage device 152/solid-state storage device 102 as a block storage device.	118	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In one embodiment, where the storage controller 152/solid-state storage device controller 202 manages multiple data storage devices/solid-state storage 110a-n, the master controller 224 divides the work load among internal controllers, such as the solid-state storage controllers 104a-n. For example, the master controller 224 may divide an object to be written to the data storage devices (e.g. solid-state storage 110a-n) so that a portion of the object is stored on each of the attached data storage devices. This feature is a performance enhancement allowing quicker storage and access to an object. In one embodiment, the master controller 224 is implemented using an FPGA. In another embodiment, the firmware within the master controller 224 may be updated through the management bus 236, the system bus 240 over a network connected to a NIC 244 or other device connected to the system bus 240.	117	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In one embodiment, the master controller 224 uses embedded controller(s). In another embodiment, the master controller 224 uses local memory such as a dynamic memory array 230 (dynamic random access memory “DRAM”), a static memory array 232 (static random access memory “SRAM”), etc. In one embodiment, the local memory is controlled using the master controller 224. In another embodiment, the master controller 224 accesses the local memory via a memory controller 228. In another embodiment, the master controller 224 runs a Linux server and may support various common server interfaces, such as the World Wide Web, hyper-text markup language (“HTML”), etc. In another embodiment, the master controller 224 uses a nano-processor. The master controller 224 may be constructed using programmable or standard logic, or any combination of controller types listed above. One skilled in the art will recognize many embodiments for the master controller 224.	116	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	Typically the data is transferred from the local bus 206 to one or more data buffers 222 as directed by the master controller 224 and the buffer controller 208. The data then flows out of the buffer(s) 222 to the data bus 204, through a solid-state controller 104, and on to the solid-state storage 110 such as NAND flash or other storage media. In a preferred embodiment, data and associated out-of-band metadata (“object metadata”) arriving with the data is communicated using one or more data channels comprising one or more solid-state storage controllers 104a-104n-1 and associated solid-state storage 110a-110n-1 while at least one channel (solid-state storage controller 104n, solid-state storage 110n) is dedicated to in-band metadata, such as index information and other metadata generated internally to the solid-state storage device 102.	112	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	Data may also be communicated to the solid-state storage controller(s) 104 from a requesting device 155 through the system bus 240, bridge 238, local bus 206, buffer(s) 222, and finally over a data bus 204. The data bus 204 typically is connected to one or more buffers 222a-n controlled with a buffer controller 208. The buffer controller 208 typically controls transfer of data from the local bus 206 to the buffers 222 and through the data bus 204 to the pipeline input buffer 306 and output buffer 330. The buffer controller 208 typically controls how data arriving from a requesting device 155 can be temporarily stored in a buffer 222 and then transferred onto a data bus 204, or vice versa, to account for different clock domains, to prevent data collisions, etc. The buffer controller 208 typically works in conjunction with the master controller 224 to coordinate data flow. As data arrives, the data will arrive on the system bus 240, be transferred to the local bus 206 through a bridge 238.	111	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In one embodiment, the solid-state storage device controller 202 includes a DMA controller 226 that controls DMA operations between the storage device/solid-state storage device 102 and one or more external memory controllers 250 and associated external memory arrays 252 and CPUs 248. Note that the external memory controllers 250 and external memory arrays 252 are called external because they are external to the storage device/solid-state storage device 102. In addition the DMA controller 226 may also control RDMA operations with requesting devices through a NIC 244 and associated RDMA controller 246. DMA and RDMA are explained in more detail below.	126	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In various embodiments, the solid-state storage device controller 202 also includes a data bus 204, a local bus 206, a buffer controller 208, buffers 0-N 222a-n, a master controller 224, a direct memory access (“DMA”) controller 226, a memory controller 228, a dynamic memory array 230, a static random memory array 232, a management controller 234, a management bus 236, a bridge 238 to a system bus 240, and miscellaneous logic 242, which are described below. In other embodiments, the system bus 240 is coupled to one or more network interface cards (“NICs”) 244, some of which may include remote DMA (“RDMA”) controllers 246, one or more central processing unit (“CPU”) 248, one or more external memory controllers 250 and associated external memory arrays 252, one or more storage controllers 254, peer controllers 256, and application specific processors 258, which are described below. The components 244-258 connected to the system bus 240 may be located in the computer 112 or may be other devices.	109	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	Beneficially, sequentially writing packets facilitates a more even use of the solid-state storage 110 and allows the solid-storage device controller 202 to monitor storage hot spots and level usage of the various virtual pages in the solid-state storage 110. Sequentially writing packets also facilitates a powerful, efficient garbage collection system, which is described in detail below. One of skill in the art will recognize other benefits of sequential storage of data packets.	107	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In a copy operation, the object index includes an entry for the original object mapped to a number of packets stored in the solid-state storage 110. When a copy is made, a new object is created and a new entry is created in the object index mapping the new object to the original packets. The new object is also written to the solid-state storage 110 with its location mapped to the new entry in the object index. The new object packets may be used to identify the packets within the original object that are referenced in case changes have been made in the original object that have not been propagated to the copy and the object index is lost or corrupted.	106	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In a read, modify, write operation, data packets associated with the object are located and read in a read operation. Data segments of the modified object that have been modified are not written to the location from which they are read. Instead, the modified data segments are again converted to data packets and then written to the next available location in the virtual page currently being written. The object index entries for the respective data packets are modified to point to the packets that contain the modified data segments. The entry or entries in the object index for data packets associated with the same object that have not been modified will include pointers to original location of the unmodified data packets. Thus, if the original object is maintained, for example to maintain a previous version of the object, the original object will have pointers in the object index to all data packets as originally written. The new object will have pointers in the object index to some of the original data packets and pointers to the modified data packets in the virtual page that is currently being written.	105	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	In one embodiment, packets are written sequentially to the solid-state storage 110. For example, packets are streamed to the storage write buffers of a bank 214a of storage elements 216 and when the buffers are full, the packets are programmed to a designated virtual page. Packets then refill the storage write buffers and, when full, the packets are written to the next virtual page. The next virtual page may be in the same bank 214a or another bank (e.g. 214b). This process continues, virtual page after virtual page, typically until a virtual erase block is filled. In another embodiment, the streaming may continue across virtual erase block boundaries with the process continuing, virtual erase block after virtual erase block.	104	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	An erase block erase command may be sent out to erase an erase block over the storage I/O bus 210 with a particular erase block address to erase a particular erase block. Typically, an erase block erase command may be sent over the parallel paths of the storage I/O bus 210 to erase a virtual erase block, each with a particular erase block address to erase a particular erase block. Simultaneously a particular bank (e.g. bank-0214a) is selected over the storage control bus 212 to prevent erasure of similarly addressed erase blocks in all of the banks (banks 1-N 214b-n). Other commands may also be sent to a particular location using a combination of the storage I/O bus 210 and the storage control bus 212. One of skill in the art will recognize other ways to select a particular storage location using the bi-directional storage I/O bus 210 and the storage control bus 212.	103	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	Similarly, a read command traveling on the storage I/O bus 210 requires a simultaneous command on the storage control bus 212 to select a single bank 214a and the appropriate page within that bank 214a. In a preferred embodiment, a read command reads an entire page, and because there are multiple solid-state storage elements 216, 218, 220 in parallel in a bank 214, an entire virtual page is read with a read command. However, the read command may be broken into subcommands, as will be explained below with respect to bank interleave. A virtual page may also be accessed in a write operation.	102	Added by DJM 3 2021	3/16/21, 12:00 AM
View Edit Delete	2380.2.01	Typically, when a packet is written to a particular location within a solid-state storage element 216, 218, 220, wherein the packet is intended to be written to a location within a particular page which is specific to a of a particular erase block of a particular element of a particular bank, a physical address is sent on the storage I/O bus 210 and followed by the packet. The physical address contains enough information for the solid-state storage element 216, 218, 220 to direct the packet to the designated location within the page. Since all storage elements in a row of storage elements (e.g. SSS 0.0-SSS 0.N 216a, 218a, 220a) are accessed simultaneously by the appropriate bus within the storage I/O bus 210a.a, to reach the proper page and to avoid writing the data packet to similarly addressed pages in the row of storage elements (SSS 0.0-SSS 0.N 216a, 218a, 220a), the bank 214a that includes the solid-state storage element SSS 0.0216a with the correct page where the data packet is to be written is simultaneously selected by the storage control bus 212.	101	Added by DJM 3 2021	3/16/21, 12:00 AM

Page 6 of 12, showing 20 record(s) out of 221 total